Green should say how many things it checked

Published November 6, 2026.

Your CI run is green. Every gate passed. One of them checked nothing.

I found two gates like that in one repo this year. Some test runners behave the same way. Each gate did its job on the inputs it got. None of them asked how many inputs it got, or whether they were the right ones.

The short answer

A gate that runs over a set must check the set before it checks the items. Is the set empty? Is it the set you meant? An empty set is not a pass. A set with the wrong members is not a pass either, even when the count looks right.

When I made one gate fail on mismatched inputs, it failed the first time it ran. That red was correct. It showed that the perf gate in moq.analyzers had not compared a single benchmark since a parameter was added to the suite.

Greens that checked nothing

The perf gate. PerfDiff is the tool moq.analyzers uses to compare benchmark runs. Its exit code is a required merge gate. It compared only the benchmarks found in both the baseline run and the PR run. Issue #1266 traced what happened when that overlap was empty. The regression checks looped over zero items. The tool logged "No regressions detected" and exited 0. A PR that broke the benchmark harness would pass the perf gate.

The coverage gate. The repo's rule, in its AGENTS.md, is 100% block coverage of modified code. Coverage collection was scoped to two assemblies. PerfDiff.dll was not one of them. On PR #1330, a PerfDiff-only change, the qlty coverage diff said: "Not applicable. There was no coverage data for the files in this diff." The gate passed. Issue #1332 records it. The fix, PR #1365, states the before and after plainly. Before, all 145 PerfDiff tests passed while the coverage report held zero PerfDiff classes. After, it held 24.

The test runner. This one is runner behavior, not an incident in the repo. pytest exits with code 5 when it collects no tests. Microsoft's migration guide says VSTest "tolerates that and exits with success" when an assembly runs zero tests. Microsoft.Testing.Platform fails the same run with exit code 8. A filter that matches nothing can be red or green, depending on the runner.

In each case the gate reported on what it saw. None reported how much it saw.

Empty and mismatched are not passes

A gate usually answers one question. Did anything fail? An empty set answers that with no. So the gate says pass.

The gate needs two questions before that one. Did I check anything? Did I check the right things? That gives four outcomes:

1. Pass: the set is the one you meant, it is not empty, and every item passed.
2. Fail: at least one item broke a rule.
3. Empty: the gate checked zero items.
4. Mismatch: the gate checked items, but not the ones you meant.

Empty and mismatch are separate failures. A count catches the first. It does not catch the second. Two sets can hold the same number of names and share none of them. The perf gate had that problem. The baseline and PR runs listed the same benchmarks under different names.

Most gates fold the last two outcomes into the first. That is the bug. A gate that cannot tell "all good" from "saw nothing" is disarmed, and it raises no alarm.

My last post made the same point about an agent. It reported PASSED with 16 of 49 files written. The validator checked format and never counted. A CI gate is the same machine with a different input.

The fix in moq.analyzers is short. PR #1309 added a check that runs before any comparison. It finds the benchmarks missing from each side and fails if either list has entries:

string[] missingFromResults = benchmarkIdToBaseResults.Keys
    .Except(benchmarkIdToDiffResults.Keys, StringComparer.Ordinal).ToArray();
string[] missingFromBaseline = benchmarkIdToDiffResults.Keys
    .Except(benchmarkIdToBaseResults.Keys, StringComparer.Ordinal).ToArray();
if (missingFromResults.Length == 0 && missingFromBaseline.Length == 0)
{
    return true;
}

logger.LogError("Benchmark result sets do not match. ...");
return false;

A separate check fails when either result set is empty. The same PR makes duplicate names fail too. Values get compared only after those checks pass.

The first red was the finding

The new check failed during work on PR #1309 itself. The perf run logged:

Benchmark result sets do not match.
Missing from results: ...Moq1400ExplicitBehaviorBenchmarks.Moq1400Baseline, ...
Missing from baseline: ...Moq1400ExplicitBehaviorBenchmarks.Moq1400Baseline(FileCount: 1), ...

Every pair was the same benchmark. Only the (FileCount: 1) suffix differed. Issue #1310 has the cause.

The perf job does not store old results. It checks out a pinned baseline commit, builds the benchmarks there, and runs them. The pin pointed at release 0.1.1, from June 9, 2025. After that release, the benchmarks gained a FileCount parameter. Every benchmark name changed.

So the baseline names and the PR names never matched. The old code took the overlap, got nothing, and exited 0. The issue puts it in one line: the gate "provided no protection for an extended period." The gate could not detect performance regressions.

The same PR fixed the baseline. It moved the pin to a commit that has the parameter. The two name sets then matched, and the gate compared real numbers again. PR #1309 merged on July 6, 2026, with the check and the new baseline together.

Two things stand out to me.

First, nobody broke the gate on purpose. A normal change to the benchmark suite disarmed it. A pinned baseline can drift out of step with the code it guards, and nothing tells you.

Second, the honest gate failed first. Expect that when you add an input check to an old gate. Treat that first red as a finding, not as a flaky check to silence.

What you can do in your setup

Start with a list. Write down every gate that loops over a set. Tests, coverage, perf, lint on changed files, security scans and schema checks all qualify.

Then work through it.

1. Print the count. A gate log should say "checked 412 tests", not just "passed". A number in the log is the cheapest alarm you can buy.
2. Fail on zero. pytest already does this with exit code 5. Do not wrap it in || true. On Microsoft.Testing.Platform, exit code 8 means zero tests. Ignore it only for the one project that needs it, not for the whole run.
3. Fail on mismatch. When a gate compares two sets, such as a baseline and a PR, check that the names match before you compare values. A matching count is not enough.
4. Scope coverage to what ships. Every assembly that ships or runs in CI belongs in the coverage filter. A file outside the filter reads as "not applicable", and not applicable reads as green.

How this could be wrong

Here is the claim in a form you can break.

• If any of the fixed gates passes when handed an empty set, the fix did not hold.
• If a count check produces more false reds than the vacuous passes it catches, it costs more than it saves. My evidence does not measure that trade.

The evidence is also narrow. It is two gates in one repo, plus documented runner behavior. It shows the failure exists and that the fix exposed a real one. It gives no rate for how often gates pass on nothing. I would like to know that number for my own repos, and I do not have it yet.

A check that cannot fail is not a check. What this adds is where to look: the inputs your gate never counted or matched.

Which of your green checks would still be green if it checked nothing?