Richard Murillo

Move the deployment failure to the keystroke

Richard Murillo — Wed, 21 Oct 2026 00:00:00 +0000

A rollout stops halfway. The validator says a configuration value is not allowed in that region. The engineer who wrote the line did it days ago, has moved on to other work, and now has to rebuild the context, find the rule, fix one value, and wait for the pipeline again. Nothing about the mistake was hidden. It sat in the source the whole time. The system just didn't say so until deploy time.

That is the problem I want to talk about: failures that are visible in source but get reported at the last possible moment.

The rule

If the cause of a failure is visible in source, the failure belongs where the source is written: in the editor and in the build. Deploy time validation stays as a safety net. It should rarely be the first place anyone hears about a mistake they could have seen while typing.

Here is the claim in a form you can test. Take the deployment failures from your last quarter and sort them into two piles: causes that were visible in source when the change was authored, and causes that were not (capacity, a dependency outage, a race). For every class in the first pile that you move to authoring time with an analyzer, a type, or a build gate, that class should stop showing up at deploy time. If it keeps showing up, the check is wrong or incomplete, and you have learned something specific about the check.

A small, shipped example

I maintain Moq.Analyzers, a set of Roslyn analyzers for the Moq mocking library. It is a small version of the same problem.

Moq builds a mock by generating a subclass, so it can only intercept members that can be overridden. This compiles without complaint:

class SampleClass
{
    int Property { get; set; }
}

var mock = new Mock<SampleClass>()
    .Setup(x => x.Property);

Without an analyzer, you find out when the test runs. Moq 4 throws a NotSupportedException whose message says that "Non-overridable members (here: ...) may not be used in setup / verification expressions" (Moq source). That is a deploy time failure in miniature: the cause was visible in the code, and the report came later, somewhere else.

With Moq.Analyzers installed, the same line gets Moq1200, "Setup should be used only for overridable members," at Error severity. The build fails at the line that caused it, with the fix in the rule doc: mock an interface, or make the member virtual. Moq1200 has shipped since release 0.0.7 and is in the current release, v0.4.2 (March 11, 2026). NuGet listed 3,834,656 downloads of the package on October 17, 2026.

A mocking mistake is low stakes. The mechanism isn't. A Roslyn analyzer runs inside the compiler and the IDE, so a rule written once is enforced on every keystroke, on every build, for every team that references the package. Nobody has to remember it.

Why this is a capability gain

A diagnostic gives the author knowledge at the moment they can act on it. The engineer doesn't need to have read the right wiki page or sat through the right incident review. The rule shows up where the decision is being made, with the reason and the fix.

The same holds for coding agents. An agent that edits a project gets the compiler's diagnostics as output it can read and act on. In my last post I argued that tools should raise what an agent can do while it has them. An analyzer at Error severity is exactly that kind of instrument: it turns a rule that lived in someone's head into a fact the agent cannot miss.

What a good authoring time check needs

Moving a check earlier only helps if engineers trust it. The checks that hold up have four things in common.

The diagnostic explains itself. It names the rule, the value it expected, and the fix. A bare "denied" is a wall, not feedback.
There is an escape hatch. Moq1200's doc shows how to suppress one violation with #pragma or turn the rule off in .editorconfig. A rule that cannot be overridden on purpose gets worked around without a record.
Severity means something. Errors block. Warnings inform. If every line has a squiggle, nobody reads any of them.
Missing input is not a pass. A check that found nothing to evaluate should say so, not report success.

Better than earlier: impossible

An analyzer catches a mistake sooner. A typed API can make the mistake impossible to write. If a deployment option only accepts the regions a service is allowed to use, the invalid value never compiles, and no check has to look for it.

Rico Mariani named this the Pit of Success in a 2003 talk to platform engineers, as Brad Abrams recorded it: "we want our customers to simply fall into winning practices by using our platform and frameworks. To the extent that we make it easy to get into trouble we fail." An analyzer at the keystroke makes trouble harder to get into. A type that refuses the wrong value makes it impossible.

Dan North put this well when he wrote about shifting testing left: it "means doing different things," and "we aren't doing that work earlier, we are setting ourselves up to never need it at all" (We Need to Talk About Testing). Analyzers are the detection half. Types are the prevention half. A good platform uses both, and keeps deploy time validation for what neither can see.

The same problem at platform scale

My team, Policy and Configuration in Microsoft Security, works on this problem at a much larger scale. Per the public job description, the team builds the shared deployment and policy capabilities that help Microsoft Security services ship safely across Azure, regions, and sovereign clouds. The work includes turning deployment, compliance, and security requirements into typed APIs, analyzers, and build gates, and moving data residency, rollout, and configuration failures from deployment time to authoring time.

The goal stated there is to make secure deployment self-service, reliable, and eventually invisible to service teams. That means clear diagnostics, safe upgrade paths, and self-service workflows, guided by support demand and adoption data. It also means a migration path from proprietary tools to standard .NET Aspire and Azure platform capabilities.

Moq1200 is one rule in one package. The interesting version of this problem is doing it for deployment policy across many services, where every rule you move to the keystroke is a class of failure service teams stop hitting.

We're hiring

I'm hiring a Principal Software Engineer in Redmond, WA to set technical direction for this work. If moving failures from deploy time to the keystroke sounds like your kind of problem, apply here: Principal Software Engineer, Policy and Configuration.

Please apply through the link rather than DMing me. If you know someone who would want this, send them the link.

Capability, not competence: agent skills as maps and GPS

Richard Murillo — Sat, 17 Oct 2026 00:00:00 +0000

Ask an agent how many of the 185 skill folders in a large public catalog contain a scripts/ directory. If it reads the listing and answers, it's estimating. A script answers by counting: 3. The model didn't lack intelligence. It lacked an instrument.

That's the whole argument in one task. I don't care whether a model can do the job after I take its tools away. I care how much it can do with them. Some skills are maps: they show the route and the branch points, a script measures what the model can't see by reading, and the model still makes the call. Some are GPS: they drive deterministic work end to end. Most real work needs both, in sequence.

The rule

Build instruments for facts a program can measure. Keep judgment with the agent when objectives or evidence are ambiguous. Automate end to end when postconditions are checkable and failure is contained. Measure task quality, cost, and recoverability against a no-skill baseline.

The common shape is a three-step hybrid:

+---------------------+     +----------------------+     +--------------------------+
| 1. Script extracts  | --> | 2. Agent judges the  | --> | 3. Script validates and  |
|    the facts        |     |    ambiguous cases   |     |    commits the result    |
|    (deterministic)  |     |    (judgment)        |     |    (checkable)           |
+---------------------+     +----------------------+     +--------------------------+

Four questions decide where each step sits:

Ambiguity. Is the objective or the evidence open to interpretation? Then the agent holds that step.
Reversibility. Can you undo a wrong result? Then automating it is safer.
Error cost. What does a wrong answer break, and who notices?
Checkable postconditions. Can a separate program confirm the result? Then a script can own it end to end.

Map or GPS: who holds the decision?

Map and GPS are the two ends of a spectrum, not two boxes. A skill that drives a task end to end can beat a map on the exact task it was built for. So the choice comes down to who should hold the decision, and who checks it.

Those are two different authorities. Execution authority is who takes the step. Verification authority is who confirms it was right. A map keeps both with the model: the model reads the evidence a script prints, compares it with what it expected, and chooses the next branch. When the route is wrong, it can notice. A GPS takes execution authority. A GPS that executes without independent checks may hide errors; a GPS with explicit checks can remain auditable. Counting, parsing, and checking a reference are good GPS work, because a script can prove the answer.

Which arrangement wins on a given task is an empirical question, and the test is below.

Capability vs competence

Capability is what a tool lets a model do while it has the tool. Competence is what the model can still do after the tool is taken away. For agents, I optimize capability.

David Krakauer's paper, Competitive and Complementary Tools (arXiv 2607.18460), gives both terms precise meanings. Competence is "the performance that remains when the tool is withdrawn." Capability "is what tools add."

Here I need to separate the model, the session, and the system. Without weight updates, using a skill does not increase the base model's learned competence; a persistent agent system can still accumulate knowledge outside its weights. When I say competence after withdrawal, I mean the base model with the skills, scripts, and external memory all removed. Measured that way, no skill I give an agent is complementary, because nothing transfers into the weights. That's fine. I'm trying to raise capability, and capability is measured with the tool present.

Where the frame comes from

Krakauer, president of the Santa Fe Institute, discussed whether AI tools are complementary or competitive on StarTalk in What Is Intelligence? With David Krakauer (Oct 2, 2026). His paper defines the split. "I call a tool complementary when competence transfers to the user and persists after withdrawal, and competitive when the user grows more dependent the more it is used and tool-free competence is compromised." An abacus or a map is complementary. A GPS is competitive. He doesn't put language models in a single column: "When its output is reconstructed and verified, it transfers a method and augments intelligence. When its answers are accepted uncritically ... it diminishes intelligence." The paper also quotes Henri Bergson's definition of intelligence as "the faculty of manufacturing artificial objects, especially tools to make tools." Tools that build tools is the goal of ai-agents.

Research on people and AI argues for human independence. The GPS for Thinking proposes a compass model so AI will "strengthen, not supplant, the student's inner compass." Not all cognitive offloading is equal separates dependent offloading from autonomous offloading. That goal is right for a student. For an agent, dependence on a good tool is the goal.

Claude skills with scripts: maps with instruments

A map with no instruments tells the model where to look but gives it nothing to look with. The script is the instrument. It does the deterministic work and reports what it found, and the model makes the call.

Anthropic's skill guidance already favors scripts. Its engineering post says "many applications require the deterministic reliability that only code can provide." Its building guide says "Code is deterministic; language interpretation isn't." That's the reliability argument. Mine adds one thing: a script that prints evidence keeps the model checking the route.

In ai-agents, skill-bootstrap produces workflow skills with a first useful action, discriminating observations, decision branches, and outcome checks (.claude/skills/skill-bootstrap/references/authoring-and-gates.md). The model still chooses the branch. ai-agents PR #6642, open as of Oct 17, 2026, adds the instrument side. Each verified item is sorted as a scaffold (prose the model reasons with), an instrument (a script that computes the answer), or both. Then bootstrap_forms.py check rejects a computable item that has no script.

The skill started as a gist I adapted in July from a prompt Fable 5 generated for u/Rodbourn, shared in an r/ClaudeAI thread started by u/oj93-rd. It asks a stronger model to write skills so smaller models can "carry this project forward without you." It also asks for instruments: "MEASURE instead of eyeball ... ship actual scripts inside the skill's scripts/ dir where they exist or where you can write them."

Skill catalogs: count is not capability

wshobson/agents has 185 skill folders, and at commit 044b4b2 (Oct 15, 2026), 3 of them contain a scripts/ directory. In ai-agents, 56 of 121 skill folders do, at commit e2b9c1e (Oct 17, 2026). Treat those counts as inventory, not evidence of capability. A count tells you how broad a catalog is. It doesn't tell you whether any one skill makes the model more capable.

The primary measure is outcome quality: the task scored with the skill and without it. Cost, latency, variance, and how often a person has to step in are guardrails. A skill that raises quality but makes runs erratic hasn't earned its place yet.

For every skill, I want five answers. Which baseline failure does it fix? What part is deterministic? Who makes the ambiguous decisions? How are failures detected? What measured improvement justifies maintaining it?

How to test whether a skill helps

The ai-agents scenario runner already compares with and without. It scores a baseline cell with no skill, a description cell, and a full cell. A skill passes only when its best cell averages at least 3.5 and beats baseline by at least 0.5 (scripts/eval/eval-rule-activation.py).

That runner measures whether the model follows the skill's guidance. This argument also needs to know whether the task came out right, so PR #6642 adds a capability eval that scores the task itself, with the instruments and without them. Its first run showed no demonstrated accuracy gain on these four tasks with Haiku. F1 went from 0.867 to 0.872 on one task and stayed at 1.000 on the other three. Cost per run went up on all four.

That doesn't mean instruments have no benefit. For a crisp question, Haiku wrote its own throwaway script, so the task got an instrument either way and a shipped script had no gap to fill. Accuracy is one outcome. The next runs also need execution time, consistency across runs, and recovery after a failure.

What I measure next

Outcome quality on an unfiltered, representative task set, plus a harder subset the model fails without help.
Repeated paired runs with confidence intervals, success criteria declared before the run, and held-out scenarios.
Four conditions with equal tool-access budgets: no skill, map, GPS, and the hybrid.
Guardrails for each condition: cost, latency, variance, intervention rate, and wrong turns the agent catches on its own.

I'll publish the numbers when I have them, including any that go against me. The skills, scripts, and evals are in ai-agents. To try it on your own repo, start with skill-bootstrap.

What task does your agent keep failing until you hand it a tool? If you've measured a skill with and without it, or have one you think should be a GPS, open an issue on ai-agents.