TaskWing GitHub
Design

Why the agents never see the tests

By Published 4 min read

Short answer

Because an agent that can see the test can make the test pass without doing the work. In TaskWing, a separate agent session writes the exam before any building starts: plain-sentence checks plus hidden tests. You approve it. Workers never see the hidden tests, and the exam runs in a clean sandbox on the joined work. In the first 3 real runs, 12 of 13 hidden checks passed, and the one failure taught us to pin the interface first.

The problem: agents grade their own homework

Ask a coding agent to add a feature "with tests", and it writes both. The tests pass. They nearly always pass, because the same agent wrote both sides with the same understanding, including the same mistakes. If it misread the task, its tests misread it too.

It gets worse under pressure. An agent that is told "make the tests pass" and can read the tests will sometimes do exactly that: special-case the input, weaken an assertion, skip a step. Not out of malice; it is the shortest path to the goal it was given.

The fix: separate who builds from who checks

  1. The lead plans. Your coding agent splits the task into pieces and pins the interface between them: routes, field names, environment variables.
  2. A separate session writes the exam. A fresh agent session, which never builds anything, writes checks in plain sentences ("an expired code is refused") and a hidden test for each one.
  3. You approve the exam. You read the plan and the checks before any agent starts. This is the cheapest moment to catch a misunderstanding.
  4. Workers build without the tests. The hidden tests never reach a worker or the lead. Workers can still write their own tests.
  5. The exam runs in a clean sandbox. On the joined branch, in a box no worker touched, plus a check that existing tests still pass.

What happened in real runs

RunAgentsTimeExam
Add a health endpoint, test repository21 min 20 s4 of 4
Bug hunt on Osprey48 min 58 s5 of 5
Alert webhook, list and disposition API on Osprey215 min 12 s3 of 4

The interesting run is the one that failed a check. Two agents wrote 2,076 lines with no merge conflict. The failed check was the exam's: the exam writer had guessed a JSON field name that nobody had fixed in the plan, and the workers had picked another.

That is the hidden exam doing its job, in an unexpected way. It showed that the plan left a gap between pieces. So we changed TaskWing: the lead now pins the interface before anyone builds, and the exam is written from the plan, after it. A check can only test names the plan has already fixed.

What we have not solved yet

  • Approval is a y/N prompt. You cannot add or edit a check yet. That is next.
  • No ruling after a failure. When a check fails, TaskWing does not yet say whether the work or the check was wrong, or explain it in plain words. Only you change a check.
  • The exam writer is also an agent. It can be wrong, as the third run showed. Your approval of the checks is the human step that keeps it honest.

The exam is the core idea of TaskWing. Parallel agents are only useful if you can trust what they hand back, and trust has to come from something other than the agents' own word.

Questions

Who writes the hidden exam?

A fresh agent session, separate from the lead and from every worker. It sees the task and the plan, not the code being written.

Can I change the exam?

Today you approve or reject it with a y/N prompt before any agent starts. Adding your own checks, which the exam writer turns into tests, is planned next.

What happens when a check fails?

The run still ends as a draft pull request, with the failed check marked, and asks you first unless you passed --yes. Only you decide whether the work or the check was wrong.

Do the existing tests still run?

Yes. The exam includes "existing tests still pass" as a check, run on the joined work in the clean sandbox.

Isn't this just test-driven development?

It borrows the idea of writing tests first. The difference is who can see them: here the tests are hidden from the agents doing the work.

Try the proof of concept

TaskWing is open source (AGPL-3.0) and early. Today it runs Claude Code agents in boat sandboxes, or in local folders to try the flow.

go install github.com/josephgoksu/TaskWing/cmd/taskwing@main
taskwing run "add a /health endpoint with a test" --provider local --no-pr