# Validation: PASSED at 16 of 49 files

An agent reported PASSED with 16 of 49 planned files written. The validator checked format and never counted. A promised-versus-delivered check catches it.

Published 2026-11-05. Canonical: https://richardmurillo.com/blog/validation-passed-16-of-49/

Your agent says the work is done and validated. The validator ran and exited clean. Only part of the work exists.

I found this failure through a retrospective in my own repo, about a commit from December 24, 2025. An agent was asked to split 18 memory files into 49 atomic files. It wrote 16, ran the validator, and reported "Validation: PASSED".

## The short answer

The validator checked the format of the files that existed. Nothing in the setup told it how many files to expect, so it could not see 33 missing. The fix is a completeness check that compares what was promised with what was delivered and fails on any gap. An instruction tells the agent to run that check. A deterministic count gives it a number it cannot talk past.

## What happened on December 24

The commit message read "Split 18 bundled memory files into 49 atomic skill files". The commit contained 16. The agent's report said `pwsh scripts/Validate-SkillFormat.ps1 - PASSED`. The [retrospective](https://github.com/rjmurillo/ai-agents/blob/main/.project-toolkit/retrospective/2025-12-24-memory-split-failure.md) records the outcome as "Incomplete split (16 of 49 planned files created)". That is 33% of the plan.

About five minutes later a second agent cherry-picked the commit and updated seven index files. Those indexes pointed at files that did not exist. The second agent had trusted the first agent's claim of completion and had not checked that the target files were there.

The CI check "Validate Memory Files" failed on PR #331 with the message "Index files reference atomic skill files that don't exist". Both commits were reverted. The retrospective puts the time from split to revert at 8 minutes.

CI caught it. The agent did not.

## Why the validator passed

The retrospective says the existing validation script "only checks format, not completeness". It validated the 16 files that existed, and all 16 passed the format check. The agent then judged its own work by the script's exit code. The retrospective's wording is that the agent self-assessed completion "based on validation script exit code, not task requirements."

The prompt for the background agent did not give an expected file count. The script did not have one either. The promise did exist. It sat in the commit message as "49 atomic skill files", and the retrospective lists the visible gap between that message and the 16 files under what helped the analysis. No script read the message. Every check the agent could run was satisfied by the work it had done. A report of PASSED was accurate about the thing it measured.

The retrospective names the root cause this way: "Background agents lack task completion verification." It also lists what it could not find. The cause of the early stop is unknown. No session log exists for the background agent's run.

## The same pattern in other sessions

The false completion was one case of a wider pattern. My [failure mode catalog](https://github.com/rjmurillo/ai-agents/blob/main/.agents/governance/FAILURE-MODES.md) draws on 50+ retrospectives from December 2025 through January 2026, plus later additions, and lists 13 patterns. One of them is False Completion Markers. Its trigger reads "Success is reported in prose instead of verified against an artifact."

The catalog's cross-cutting theme opens with a sentence I now work from: "Trust-based compliance fails at scale." Instructions that ask an agent to "remember", "verify" or "check" without an observable artifact "succeed briefly and degrade as context grows". The replacement is "a blocking gate that produces an artifact a tool can inspect."

Instructions work for a while. A January 2026 [session protocol analysis](https://github.com/rjmurillo/ai-agents/blob/main/.project-toolkit/retrospective/2026-01-09-session-protocol-violation-analysis.md) shows where they stop. In fresh sessions the agent had "100% Serena activation compliance". Serena is the semantic code tool the session protocol tells it to activate. The violation it analyzed happened in a continuation session after context compaction, where the agent skipped the start-of-session steps. The analysis says the work "was NOT blocked by missing protocol steps".

The same analysis tested a git pre-commit hook that checked for a session lock. Without `--no-verify` the hook blocked the commit. With it, the hook never executed and the commit succeeded. A check the agent can skip behaves like an instruction. My [previous post](https://richardmurillo.com/blog/instruction-asks-hook-checks/) covers the case for hooks and where my evidence for them stops.

## The completeness check

The fix went into the QA agent's prompt in April 2026 as a [Completeness Verification section](https://github.com/rjmurillo/ai-agents/blob/main/src/claude/agents/qa.md#completeness-verification-mandatory) marked mandatory. It opens with "Before reporting validation results, verify completeness independently. Format checks alone do not verify scope."

It asks three questions. What was promised. What was delivered. Compare. The worked example mirrors the December failure: "If the promise was 'update all 49 templates' and only 16 exist, validation = FAIL regardless of whether those 16 are correct."

```text
Promised: [list from issue/delegation]
Delivered: [list from workspace]
Gap: [missing items, if any]
Result: PASS | FAIL
```

The prompt requires that block above the verdict, so the verdict is "backed by that reconciliation, not asserted alone." If the agent cannot find what was promised, it must stop with `work_finish(blocked, "Cannot verify completeness without requirements")`. The prompt's own rationale line cites the 16 of 49 incident. Its core mission section says the rest in two sentences: "Validation is not passing a test suite. Validation is verifying what was supposed to be built actually got built."

The retrospective proposed the script-side version too. Its pattern has four steps. Document the expected output count in the commit message. Assert that the count of created files equals the expected count. If they differ, fail with an error that shows actual against expected. Commit only when the count matches. Its checklist writes it as "Expected file count: 49" next to "Actual file count: 49".

The instruction makes the agent ask the question. The count gives the answer a denominator. Together they make the report something a tool can inspect. A test in the repo, [`test_qa_prompt_contract.py`](https://github.com/rjmurillo/ai-agents/blob/main/tests/build_scripts/test_qa_prompt_contract.py), fails the test suite if that section is removed or if copies of the QA prompt drift apart.

Matt Kotsenas describes the same gap from the orchestration side in [The Control Paradox](https://matt.kotsenas.com/posts/the-control-paradox/). He writes that an agent asked to "do this in 10 places" does it in 5. His fix is to get "task state out of the chat history and into something durable". The promise has to live somewhere the check can read it.

## What this evidence covers

It covers one incident, documented in one retrospective, and a catalog that generalizes from many. It does not give a rate. The January analysis lists compaction-related violations as "1 observed (likely underreported)", and it says it is unknown how frequent the violation is. The catalog also cites a 95.8% Session End failure rate, 23 of 24 sessions on one day (2025-12-20) on one branch. Treat it as a direction.

The QA check is itself an instruction in a prompt. The test shows the instruction is present. Whether an agent obeys it is untested here. I have not replayed the December 24 split through the QA agent, so I do not claim it would report FAIL at 16 of 49. If that replay reports PASSED, this post is wrong about the fix. The retrospective's count assertion is the deterministic half, and I have not shown it running against this case either.

The retrospective also leaves open why the agent stopped at 16. A count check would catch the stop. It would say nothing about the cause.

## What you can do in your setup

Write the denominator down before the work starts. Put the expected count in the task file, the issue or the commit plan, anywhere a script can read it.

Add one script that compares it with what exists on disk, and fail with both numbers. Run it where the agent cannot skip it, such as CI.

Ask your validating agent for a Promised, Delivered, Gap block, and make the verdict depend on it. Tell it what to do when the promise is missing.

Check the handoff too. The second agent in December updated the indexes before it confirmed the target files existed. The retrospective calls that the Index-First Anti-Pattern. Confirm that the files exist before anything links to them.

Then test the check. Take one partial result from your own history and run it through. If it reports PASSED, fix the check before you trust it.

What is the last number your agent reported that nothing in your setup counted?