A nine-minute recap of a talk by Lance Martin, who works on Anthropic’s applied AI team, has been making the rounds this week. The talk was given at AI Engineer and posted in early November, and its pitch is right in the first clip. “Many times you’re not actually on this frontier,” Martin says, meaning the curve where paying more for a model buys better answers. “Your application is actually sitting inside this frontier… because of minor misconfigurations.” The recap boils it down to seven moments and six slash commands you can run tonight.
I haven’t watched all 64 minutes. What I have is the recap’s clips in Martin’s own voice, the YouTuber’s narration around them, StartupHub.ai’s write-up of the talk, and the primary sources the commands rest on. The commands are listed in Anthropic’s Claude Code changelog, and they run instructions that ship inside Claude Code’s bundled claude-api skill, which anyone can read. Most of the advice holds up. Two numbers need context, and one safeguard is weaker than the recap makes it sound.
The commands exist, with slightly different names
The auto-captions mangle every command, so here they are as the changelog spells them. Claude Code 2.1.283 “added /doctor prompt-audit (also /checkup prompt-audit) to audit your CLAUDE.md files, skills, agents and commands for prompting patterns written for older models.” The API-side version is /claude-api prompt-audit. The others are /claude-api cost-optimize, /claude-api build-eval, /claude-api hillclimb (one word) and /claude-api managed-agents-onboard. Typing what the captions say won’t work.
The prompt-habit advice is the least controversial part. Martin’s list of things to stop writing is “think step by step,” “double check your work” and “be maximally thorough,” and the audit’s own pattern table agrees nearly word for word. It flags “think step by step” as an incantation that is “redundant at best” on models with adaptive thinking, and its suggested fix for “Be thorough. Do not be lazy. Do not stop early.” is to delete the line. The reason it gives for padding costing money is concrete: duplicated rules “make the model spend effort reconciling wordings,” and “bulk also directly inflates adaptive-thinking spend.”
The recap’s advice for people who don’t code is to delete any line that says “think step by step.” The audit itself is more careful. It treats dated phrasing on its own as “a flag-only signal (low confidence),” and a later release changed it so that “thinking keywords that Claude Code documents are kept.” Delete the incantations, then, but leave alone the ones your own tools document.
The 72% was measured on untuned setups
This is the line that got clipped and shared. “Myself and some folks on our applied AI team ran this command on a bunch of internal benchmarks that had kind of default harnesses. We found opportunities to reduce cost by up to 72% with no meaningful change to performance.” StartupHub reports the same figure.
The qualifiers are in the sentence. “Up to” makes 72% the best case. “Kind of default harnesses” means the setups started out naive. The recap describes the range as “about a quarter to almost three quarters of the bill.” I couldn’t find that range anywhere outside the recap, so I’d treat it as the YouTuber’s paraphrase. If you have already put your stable content first and your volatile content last, you have probably taken most of what this command would find.
The mechanism is simple enough to check by hand. Caching is a strict prefix match. Anthropic’s caching guide says “any byte change anywhere in the prefix invalidates everything after it,” and the classic culprit is the one Martin names, a timestamp at the top of the system prompt. The recap leaves out the part that makes this worse than a missed discount. A cache write costs 1.25 times the base input price on the five-minute cache. If your prefix changes on every turn, every turn is a fresh write, and you pay more than if you had never turned caching on.
“About a tenth” was last generation’s price
The recap says that when the repeated instructions come from the cache, “you pay about a tenth of the price.” That was the standard rate, and it still is on many models. Anthropic’s pricing page lists cache hits at 0.1 times base input on Opus 4.8, Sonnet 5 and Haiku 5.5. On the current Opus and Sonnet, though, it’s half that. “On Claude Opus 5.5 and Claude Sonnet 5.5, a cache hit costs 5 percent of the standard input price.” On Fable 5.2 it is 2.5 percent.
So the advice errs in the safe direction. On the current Opus, Sonnet and Fable models, a cache miss costs 20 to 40 times a hit, not 10. Every byte that breaks the prefix costs proportionally more than the talk’s rule of thumb implies. I made a similar point in September about how much of an agent’s bill is cache reads. That share is the line item most price comparisons skip, and it’s the one this talk is about.
Smart model, low effort, then measure
The fourth moment, which the recap says sits 39 minutes in, during the Q&A, is about the effort setting. “There’s generally an intuition that moving to higher models at lower effort is often cheaper and more performant than lower models at high effort. Try low to start.” His summary is “very very smart model not trying too hard.”
Anthropic’s own guidance says much the same. Its cost guide says that in Anthropic’s research and knowledge-work runs, “low gave up a few points for a third to a half off cost per task, medium matched high’s accuracy for noticeably less, and high bought nothing measurable over medium.” It also says the effect isn’t guaranteed. On one coding benchmark, “Claude Fable 5.2 at its default effort cost more per solved task than Claude Fable 5.1.” I spent a whole post in September on how effort level names don’t mean the same amount of thinking from one model to the next. That’s one more reason to test on your own task instead of carrying a setting over.
A viewer pushed back in the comments that for some coding jobs, Haiku on max effort came out cheaper. The arithmetic makes that easy to believe. Haiku 5.5 lists at $0.10 per million input tokens against $4 for Opus 5.5, a 40-fold gap. Haiku can think much harder and still cost less. Whether it gets the answer right is a separate question, and the only way to find out is to count cost per completed task on your own work, which is where the eval comes in.
The recap credits Martin with a support-ticket example in which accuracy rose and cost per ticket fell by about 90%. The recap and StartupHub disagree on which models were involved. StartupHub describes a 50-ticket eval in which moving to a newer Opus model “saved money and slightly raised quality,” with an audit adding a further gain. I’d cite the direction, not the model names.
Pass or fail, and how many tickets
Martin’s grading advice is crisp. “Don’t use arbitrary scales… grade it from 0 to 5. It’s very ambiguous. Make it crisp and simple like ideally a binary classification.” The recap adds two more rules: have two models grade the same answers and treat disagreement as a sign the rubric is broken, and drop questions nobody can answer, because “no model can actually achieve it.”
All of that matches the build-eval instructions, which ask for “concrete, checkable claims… rather than vague scales (‘rate helpfulness 1-5’).” It also fits research on LLM graders. A 2024 study in npj Science of Learning found that even free models graded well against criteria once the criteria were detailed, and its authors put domain understanding ahead of model complexity. The same instructions also add a caveat the talk skips. When what you care about is a tradeoff, like “shorter is better, but not at the cost of completeness,” the instructions prefer a continuous measure, because “a binary grade throws away the shape of the tradeoff.” Pass/fail also has a statistical cost. A binary score carries less information per case, and the eval audit lists “a pairwise or continuous metric instead of a binary one” as one way to lower the noise.
The eval audit gives a rule of thumb for that noise. For a pass rate, the 95% band on a before-and-after difference is “roughly 1/sqrt(n·R),” where n is the number of cases and R the repeats: “25 cases × 2 reps ~ ±14 points, 100 × 2 ~ ±7.”
Apply that to the recap’s best story. The YouTuber says he ran the hill climb in September on “a small support ticket app” and went from 74% to 100% “on tickets it had never seen,” while cost fell from about 3 cents to a third of a cent per ticket. He doesn’t say how many tickets. The audit also warns that “if the baseline already scores ~95%+, the eval cannot discriminate at the top,” and that near the ceiling the last points “may no longer measure what the eval was built for.” A perfect held-out score on a small set fits a real improvement. It also fits a test set that was too easy. I’d want the case count before drawing conclusions from either number.
The held-out pile does get used
Moment six is the overnight loop, where you let Claude improve the prompt against the eval while you sleep. Martin names the trap himself. “You spend a lot of tokens hill climbing… It looks really good after hill climbing. You run it in production and performance is not better at all.” The recap’s version of the defense is that the command “splits your tests into two piles. Claude studies one pile and never sees the other.”
The hillclimb instructions do split the set, and they’re strict about who reads what. A fresh analyzer each round reads only training transcripts, and “the outer session does not read transcripts itself.” The test set is “never opened by the analyzer.” The same paragraph also says what test is for: “its score picks the winning round and is the headline.”
That’s a reasonable design, but “never sees” overstates it. A number that chooses between rounds is a number the process has looked at. Run eight rounds and keep the best test score, and part of that score is the luck of the draw. Statisticians call this adaptive data analysis. Dwork and colleagues, writing in Science in 2015, describe data analysis as “an intrinsically adaptive process, with new analyses generated on the basis of data exploration, as well as the results of previous analyses on the same data,” and they built their reusable holdout method because the plain kind can’t be reused freely. The skill’s authors know this. For sets of about 150 cases or more, they suggest a third validation slice to pick the winner, with test held back “until the end.” They also say that for most evals “the extra bookkeeping isn’t worth it.” On a small ticket set, I think it’s worth keeping a few dozen fresh tickets that the loop never scores at all, and checking the winner on them once.
Keys the agent never holds
The last moment is about agents that run for days. Martin co-wrote Anthropic’s April engineering post, “Scaling Managed Agents: Decoupling the brain from the hands,” and the recap’s version matches it. The session is “the append-only log of everything that happened,” and “because the session log sits outside the harness, nothing in the harness needs to survive a crash.” On credentials, the post says “auth can be bundled with a resource or held in a vault outside the sandbox,” and “the harness is never made aware of any credentials.”
The recap says the agent “can use them, but it can never read them.” The skill’s own description is slightly more careful. Vault secrets are “substituted at egress, so the sandbox holds only a placeholder unless a service sends the secret back.” That last clause is a real exception. An API that echoes your key back in an error message puts it in front of the agent.
What I’d run, in order
The recap’s closing summary is about right, and the order matters. First, run the prompt audit and delete what it flags with high confidence. Then make the prefix byte-stable and check cache_read_input_tokens to confirm it, because on Opus 5.5 that’s a 20-fold price difference. Next, try the strongest model at low effort before reaching for a cheaper one, and compare cost per completed task, not per token. Build a pass/fail eval with enough cases that a real gain shows up above the noise, then let the hill climb run. When it’s done, check the winner once on tickets the loop never scored. Every step after the first two depends on that eval, and so does every number in the recap.
References
- Martin, L. (2026). Your Old Prompts Are Making Claude Worse. Talk at AI Engineer, November 2026. Quoted from clips in the recap below. I have not watched the full talk.
- Recap of Lance Martin’s AI Engineer talk. YouTube video, about nine minutes, November 2026. Quoted from its auto-caption transcript.
- StartupHub.ai (2026). Your Old Claude Prompts Are Costing You Twice. 7 November 2026.
- Anthropic (2026). Claude Code changelog, entries for 2.1.221 (
prompt-auditsubcommand), 2.1.283 (/doctor prompt-audit,/checkupalias) and thecost-optimize,hillclimbandmanaged-agents-onboardentries. Accessed 11 November 2026. - Anthropic (2026). Claude Code bundled
claude-apiskill:shared/prompt-audit.md,shared/prompt-caching.md,shared/cost-optimization.md,shared/evals/build-eval.md,shared/evals/eval-audit.md,shared/evals/eval-hillclimb.md. Claude Code 2.1.322, accessed 11 November 2026. - Anthropic (2026). Pricing. Claude Platform documentation, accessed 11 November 2026.
- Martin, L., Cemaj, G. and Cohen, M. (2026). Scaling Managed Agents: Decoupling the brain from the hands. Anthropic Engineering, 8 April 2026.
- Dwork, C., Feldman, V., Hardt, M., Pitassi, T., Reingold, O. and Roth, A. (2015). The reusable holdout: Preserving validity in adaptive data analysis. Science 349(6248), 636-638.
- Zhang, D., Boey, M., Tan, Y. Y. and Jia, A. H. S. (2024). Evaluating large language models for criterion-based grading from agreement to consistency. npj Science of Learning 9, December 2024.