AI Coding Agent That Runs Tests: Check the Evidence

A freelancer in Pune asks an AI agent to add a discount rule to a client's checkout code. The agent replies: "Done. I ran the tests and everything passes." She merges, deploys, and two hours later the client messages her about a negative cart total.

Nothing in that reply was necessarily a lie. But nothing in it was evidence either. "I ran the tests" is a sentence. A test run is a command, an exit status and a list of results you can look at.

If you want an AI coding agent that runs tests, you should be able to confirm it in under two minutes, in your own terminal, without trusting the agent's summary. This post gives you the method. It separates five things people blur together: test code written, tests executed, exit status, failing tests, and reruns after a fix. It also explains why a benchmark score or a browser replay tells you little about your own task, and where OxCode fits in the check.

Why 'it ran the tests' is a claim, not proof

A language model produces text. When it writes "all tests pass," that text can come from a real command it ran, or it can come from pattern completion: tests usually pass at the end of a coding story, so the sentence feels right. From the outside, both look identical in a chat window.

So treat the sentence as a claim. A claim needs a receipt. For a test run, the receipt is concrete: the exact command that was executed, the working directory it ran in, the output it printed, and the exit status it returned. If you can't point to those four things, you have a story, not a result.

This matters more when you work alone. A freelancer or indie hacker has no reviewer catching a fake green. A student has no senior engineer asking "which command did you run?" You are the review process, so you need a cheap, repeatable way to doubt the sentence and still move fast.

The fix isn't to distrust the tool. Make the evidence visible and check it the same way every time. A good agent makes this easy: it runs commands in a sandbox, shows you what it ran, and fixes its own bugs before it says the task is done. OxCode is built around that loop, and its approval-first commands show you what is about to execute instead of letting you find out afterwards.

Here is the position of this post: a claim about testing is only as good as the five things you can inspect behind it. The next section names them.

Five things people confuse: written, executed, exit status, failures, reruns

Most confusion comes from collapsing five separate facts into one phrase. Pull them apart.

1. Test code written. The agent created or edited a test file. This proves only that text exists on disk. A test that asserts expect(true).toBe(true) is test code. It tests nothing.

2. Tests executed. A test runner actually ran, such as pytest, npm test or go test ./.... Written is not executed. Plenty of agents write a test file and never invoke the runner, or invoke it against the wrong folder.

3. Exit status. Every command returns a number. Zero means success, non-zero means failure. This is the most honest signal in the whole loop, because the runner produces it, not the model. Run echo $? right after the command (or echo %ERRORLEVEL% on Windows cmd) and you have it.

4. Failing tests. If the exit status is non-zero, which tests failed and why? A summary line like 2 failed, 14 passed is the evidence. "Some tests had minor issues" is not.

5. Reruns after a fix. If something failed, did the agent change code and run the same suite again, and did the second run go green? A fix with no rerun is a guess.

A concrete before and after. Weak report: "Added tests for the coupon logic, all passing." Strong report: "Added test_coupon.py (6 cases). Ran pytest tests/ -q. First run: 1 failed (test_expired_coupon, expected rejection, got discount applied). Fixed the date comparison. Reran: 6 passed, exit 0."

The second version names the command, the failure, the fix and the rerun. You can reproduce every line of it. Hold any agent to that standard.

A verification checklist you can run in your own terminal

You don't need special tooling. Open the integrated terminal in VS Code, Cursor or whichever VS Code-based editor you use, and run this checklist after any task the agent calls done.

Step 1: Ask for the command. What exact command did it run? Copy it. If it can't tell you, that is your first red flag.

Step 2: Run it yourself, from a clean state. Same command, same folder. Don't rely on its output. For a Node project that might be npm test; for Python, pytest -q. Your terminal result is the one that counts.

Step 3: Read the exit status. Right after the run, check echo $?. Zero is a pass. Anything else is a fail, whatever the summary text said.

Step 4: Count the tests. Note how many ran. If the suite had 40 tests yesterday and has 6 today, something was deleted or skipped. Look for skipped, xfail, and 0 tests collected.

Step 5: Break something on purpose. This is the best check there is. Change one line of the new code so it is wrong (flip a > to <), and rerun. A real test fails. A decorative test stays green, and now you know it protects nothing. Revert the change.

Step 6: Check the diff for test tampering. Run git diff on the test files. Did the agent loosen an assertion, delete a case, or add a skip marker to get to green?

Step 7: Rerun after any change. If the agent fixed something after your check, run the whole suite again.

This takes about two minutes on a small project. To practise on a real task, the first-task walkthrough for a free AI coding agent in VS Code is a low-risk place to run these steps once and build the habit.

What published benchmarks and browser replays do and don't prove

Benchmarks are useful. They are not a receipt for your repository.

Take SWE-bench Verified. A subset of the broader SWE-bench benchmark (which asks a model to generate patches evaluated against tests across 2,294 issues from 12 Python repositories), it consists of 500 engineer-confirmed solvable problems. OxCode resolved 410 of 500, one attempt each, which is 82.0%, and Terminal-Bench 2.1 has 73 of 89 tasks solved, also 82.0%. Public run records sit at oxcode.ai/benchmarks. Those records are worth reading because you can inspect them instead of taking a headline number on faith.

But be precise about what the number measures. It measures performance on that fixed set of tasks, in that harness, in those repositories. It says nothing about your Django monolith, your half-migrated React app, or the flaky integration test your client's team never fixed. Your task has your dependencies, your test conventions and your quirks.

Browser replays have a similar limit. A recorded session, where an agent clicks through a UI and a green banner appears, shows that one path worked once. It doesn't show which tests exist, whether they cover the change, or what the exit status was. A replay is a demo. A demo is evidence that something can happen, not that it happened on your code.

Use benchmarks to shortlist a tool, then use the checklist above to judge a specific task. Benchmark scores answer "is this agent capable in general?" Your terminal answers "did it do the work on my project, today?" Only the second question decides whether you can merge.

Red flags that a green result isn't real

A green result can be false in quiet ways. Scan for these patterns, roughly in order of how often they show up.

The report has no command in it. "Tests pass" with no pytest or npm test line anywhere. Treat it as unverified until you run something yourself.

The test count dropped. Fewer tests than before is the cheapest way to turn red into green. Compare counts against your last known run.

Assertions got weaker. Look in git diff for toEqual changed to toBeDefined, a specific value replaced with any(), or an expected exception removed.

New skip or xfail markers. @pytest.mark.skip, it.skip(, xit(, @Ignore. One of these on a test that used to run is a flag.

Mocks swallowed the thing under test. If the new test mocks the very function you changed, it verifies the mock. Look for tests where nearly every dependency is stubbed.

The run was in the wrong place. A suite run in a sibling folder, or with a different environment, or with --passWithNoTests, can exit 0 while running nothing.

It never failed. An agent that writes a feature plus tests and reports a clean first run on a non-trivial change deserves one extra look. Real work usually produces at least one failure that gets fixed, which is why the rerun step matters.

Exit zero, but the output has errors. Some scripts print tracebacks and still exit 0 because of how they are wired. Read the output, not only the number.

When you hit one of these, don't argue with the summary. Rerun the suite, do the break-it-on-purpose check, and read the diff. For the cases that do fail, the guide to debugging AI-generated code in VS Code before you ship it walks through a systematic method.

Run the check on your own task

Reading a checklist is easy. The habit forms when you run it on code you care about.

Pick one small, real task from your current project: a validation rule, a bug with a known reproduction, a utility function that has no tests yet. Give it to your agent and ask it to plan, write the change and the tests, run them, and report the command and the exit status. Then ignore its summary and run the seven steps yourself.

OxCode is designed for exactly this loop. It plans, writes, tests and debugs code, runs what it writes in a sandbox, and catches and fixes its own bugs before calling a task done. Its Debug and Plan modes give you somewhere to go when the first run fails, and approval-first commands keep you in the loop on what gets executed. Whatever the tool tells you, your terminal gets the last word.

Start with the first-task guide to get set up in VS Code or a VS Code-based editor, and keep the debugging guide open for the first time something goes red.

Pricing details, if you are comparing options, are laid out in the rupee pricing guide.

Conclusion

Stop accepting "all tests pass" as a status update. From your next task onward, require a command, an exit status, a failure count and a rerun, and verify them yourself in the terminal before you merge. My prediction: within a year, the agents worth paying for will be the ones whose reports you can reproduce line by line, and the ones that only summarise will lose developers who have been burned once.

Install OxCode in VS Code, Cursor or Antigravity, give it one real task from your own repo on the Free plan, and run the seven-step checklist on what it hands back. If the evidence holds up in your terminal, you have your answer. If it doesn't, you found out in two minutes instead of after deployment.