The Bottleneck Moved: Some Notes on AI Engineering, from Code to Product

AI changes quickly, and keeping up has become work of its own. What follows is my reading of that constraint and where it sits in November 2026, written from day-to-day work on AI-first products rather than from a lab; parts of it may look wrong next month. Building that way changed my process more than any methodology I have adopted: loops shorter than a sprint, a plan that moves inside the loop, prototypes worth throwing away, and a perspective that has to climb out of the code and look at the customer, the data, and the operations.

The article mixes two subjects that are easy to confuse: AI as a tool for a software engineer, and AI-first software, where models, harnesses, and agents are the product a customer uses. Both have been my work over the past year, a year in which software development and product merged in ways that were not possible before. What follows is not the life of every developer: it is one engineer’s opinions from building AI-first products, which is why it reads as notes rather than as one argument. Evaluation, harnesses, efficiency, and autonomy each deserve an article of their own.

If there is a short version, it is that capability stopped being the constraint I plan around. Verification became the constraint, and trust in what may act on its own sits behind that. Generating code became cheap; making it survive production, and deliver something someone will pay for, is not.

A developer who stays inside the code castle, judging diffs and never asking whether the thing should exist, is comfortable and replaceable. Am I solving a real problem here, or implementing a card and calling it done? Being a code typist is not a moat; it is the first part of the job a model already does faster. Writing code was never the essence of the work, solving problems was, and if AI takes the typing, it gives back the time to see the whole picture: the customer, the operations, the cost of being wrong. That is the developer I would bet on, the one who knows which problem deserves the code and can tell whether the answer is worth shipping. No model is going to make that call.

Where things stand

These are the judgments I would defend in a design review today. Taken together they are the argument; what follows is the working out. The questions behind them are the ones I ask of a pull request and of a product alike: where the capability is, what a completed task costs, and how much of it can be left unsupervised.

The gain is in the loop, not in the answer. In code, a single call writes something that looks like code; the loop is what writes something that can be checked against a test, a type checker, or a running service. In a product, the difference shows up as repeatability: a demo runs once with a person guiding it, while a feature has to work when nobody is watching. The loop narrows that gap without closing it, because it can only check what someone thought to check.

The harness is part of the model you actually use. In code, the same model under different tools, instructions, context assembly, and validation behaves like a different colleague. In a product, the harness is the product: it decides what the model can touch, what it remembers, and what it is never allowed to do. The model is rented; the behavior is built.

Writing is no longer the bottleneck; verification is. In code, reviewing the diff, deciding whether the tests prove the right thing, and judging the architectural fit now cost more than producing the change. In a product, the same shift shows up as an acceptance question: who confirms the output is right, and what happens when nobody does. Seniority moves from producing to judging in both.

Coding is the baseline; the value is in the process. In code, generation is the floor every provider competes on, so the value moved to what surrounds it: tests, types, review gates, CI checks, and observability. That ecosystem is what catches generated code before it reaches production, and code now appears at a speed no team could review at the old rate. Building the apparatus that proves the output is safe is the real work. In a product, the customer is not buying a model; they are buying the disappearance of a manual step, which is why the products that last take an expensive routine and make it defined and verifiable.

Long tasks remain the weakest case, and the clearest opening. In code, a session spanning hours loses state, and the agent declares completion early. In a product, the same failure looks like a support agent that forgets the ticket, or a pipeline that reports success on a half-processed batch. The answers so far are harness answers, incremental progress, external state, and tests, and none of them is finished. A product that holds state, resumes where it stopped, and can prove what it actually completed is already ahead, whatever approach it took to get there.

Benchmarks give direction; your tasks give the answer. In code, a benchmark dashboard narrows the field to three or four candidates. In a product, the deeper problem is non-determinism: the same input can take different paths, so behavior cannot be predicted from a leaderboard or from a single run. Testing on real cases and adapting the surrounding system is how you get close, and nothing is ever tested exhaustively before release, because perfect coverage costs the time the market will not give you. Evaluating behavior in production and fixing fast is part of the process now.

Cost is a per-task decision, not a launch criterion. In code, the choice is often not the best model but the good-enough one for each step. In a product, the same decision is unit economics, and it is a trap to make it early: projections rarely match real usage, and trimming cost before anyone has used the product spends time on a number that will be wrong. Ship it, watch what it actually costs, and optimize once a user has shown the thing is worth paying for.

Autonomy is a boundary someone sets, not a property of the model. In code, an agent that reads a repository and proposes a diff carries a different risk from one that pushes to main, and the difference is a permission, not a better model. In a product, the line shows up in plain verbs: draft the reply or send it, suggest the refund or issue it, flag the alert or silence it. Capability does not move that line. The product has to decide where it sits and enforce it.

One caveat about the third judgment, because it is the easiest to repeat and the hardest to measure. In 2025 METR measured a 19% slowdown across 16 experienced developers using the tools of the time, and the same developers went on believing they had been about 20% faster. In February 2026, with 57 developers and more than 800 tasks, the estimate turned into an 18% speedup for those who returned to the study, with an interval from −38% to +9%, and the team concluded the data had stopped being a reliable signal: between 30% and 50% of participants stopped submitting tasks they would have done without AI. If seniority moves to judging, the distance between what is felt and what is measured becomes the judge’s problem, not a footnote for whoever does the measuring. Sources: METR, 2025 and METR, 2026.

From here the article moves through the history that produced these judgments, the model and the harness that carry them, and the decisions they force on a team building either a codebase or a product.

From answers to agents

The first version most of us met was a model, not a product. GPT-3 arrived in 2020, first as a paper and then as an API behind a waitlist, and what it did was strange enough to hold attention: it completed text from a prompt in a register close enough to human writing to be useful for something. Nobody I knew could say what. It was a capability in search of an application. Source: GPT-3 paper.

ChatGPT, at the end of 2022, did not add capability; it added a place to put it. A text box, instructions, a conversation, and a model answering from its training data — a harness so thin it barely qualifies as one. It was also an immediate hit, which says something about how much of adoption is the interface rather than the model.

The thinness showed in the failures. A model that answers from memory has no way to check itself, so being wrong is not an edge case; it is the normal outcome of confidence plus missing information. In February 2023, Google published a promotional demo in which Bard claimed the James Webb telescope had taken the first image of an exoplanet — a claim a single search would have corrected, issued by the company whose product was search. Alphabet lost about $100 billion in market value in a day. Source: Reuters. A year later the Bard name was retired in favor of Gemini, presented as aligning the product with its model family: accurate, and also the kind of explanation a company offers after a launch it would prefer to forget. Source: Google.

What changed after that was mostly not the model’s intelligence. It was what the model could reach.

• Search before answering. A claim can be grounded in a page instead of in memory, and the answer can point at what it read.
• Calling code. Asked for a calculation, the model writes a short script, runs it, and returns what the interpreter produced. Arithmetic stopped being a prediction; the result is exact because the model is no longer the thing that computes it.
• Calling tools in general. The model requests an action, observes the result, and continues. That is the difference between describing a refund and issuing one.
• A standard for the connections. MCP, open-sourced by Anthropic in November 2024, replaced bespoke integrations with one protocol, so a calendar, a mailbox, a repository, or a bank’s API can be exposed to a model the same way. Source: Anthropic.

The other change happened inside the model. Reasoning used to be a prompt technique: tell it to work step by step, ask whether it is sure, ask it to argue the opposite, then iterate until the answer held. Careful users got thinking by making the model talk longer. The reasoning models that arrived from 2024 onward moved that discipline into training: the model spends tokens on its own argument, tries a strategy, notices it is not working, and revises before answering. Source: OpenAI.

That is still a probability, not a proof. The model iterates until it judges its answer likely enough to be right, which is not the same as being right, and the failures that remain are more interesting than “it does not know”.

Tool use and reasoning together are what turned this into a product category rather than a demo: Cursor, Windsurf, Claude Code, Codex, and everything modeled on them. In each case the model reads, edits, runs, checks, and reports, instead of producing text for a person to paste.

Agents move that capability across a whole workflow. In this article, agentic describes a system that receives a goal and can decide, within certain limits, which steps and tools to use to achieve it. A single model call produces an answer or a piece of code. An agent uses that answer to choose a tool, observes the result, updates its context, and takes the next step, until the task is complete, a limit is reached, or a human is needed.

Given a bug, a coding agent might:

1. Search the repository for where the behavior is implemented.
2. Read the related code and tests.
3. Propose a fix.
4. Run the available tests.
5. Analyze a failure and adjust the implementation.
6. Present the diff and explain what was validated.

Depending on its environment and permissions, it can also consult documentation, call APIs, use a browser, or reach other systems. Every step can fail: it may look in the wrong place, misread a rule, or declare the work complete without testing the behavior that mattered. The loop of investigate, implement, verify, and correct is what creates the chance to catch those failures before delivery. It is also where the work either becomes trustworthy or quietly accumulates debt.

Seen from outside the editor, the change of scale matters more than the change of speed. With a model that reasons, searches, and acts on other systems, the cost of a step stops being the cost of implementing it. A process that used to need a project and a quarter — intake of documents, reconciliation between systems, triage at three in the morning, follow-up nobody has time for — can be stated as a goal with a definition of done and turned into software. What a team can attempt stops being bounded by how much each step costs, which is where the sense of limitlessness comes from. None of it is the model being magic; nothing in a model invents value on its own. The constraint moved, and it moved to the harder place: not whether a problem can be solved, but which of the solvable problems is worth solving, and which of its features a customer will pay for. That decision has not been automated.

Model and harness: different parts of the same system

To understand where the progress comes from, it helps to separate the model from the environment around it. The term settled as Agent = Model + Harness: everything that is not the model is the harness, and the harness is what gives the model state, tools, and verification loops (LangChain). Böckeler starts from that definition and narrows it for people who use a coding agent, separating the guides that come before an action from the sensors that correct it (Böckeler).

The language model receives context and generates responses. In a workflow with tools, it can produce a structured tool call and then interpret the result.

The harness is the software layer that organizes that interaction. Depending on the implementation, it can:

• Provide tools to read and edit files, run commands, and access services.
• Assemble the context sent to the model, including instructions, history, and tool results.
• Manage the loop between model calls and tool execution.
• Apply permission policies, limits, sandboxing, and error handling.
• Preserve or summarize information for tasks that span multiple context windows.

Claude Code, Codex, and other development tools offer combinations of these capabilities. The exact behavior depends on the product and its configuration.

The distinction matters because the result does not depend on the model alone. The same model behaves differently under different tools, instructions, context, and validation, and the difference is large enough to measure: on the same 25 tasks, with the same provider and the same tools, eight harnesses took the same model from 68% to 88% success (Marmelab). On Terminal Bench 2.0, the same Opus 4.6 inside Claude Code scores well below what the same model delivers in other harnesses, and changing only the harness moved one agent from Top 30 to Top 5 (LangChain).

This is why I now evaluate agents as whole systems: model, harness, tools, permissions, context, and execution environment. A model comparison that ignores the harness measures something nobody runs.

The limitations of the model are also where the harness earns its keep. In experiments with long-running agents, Anthropic observed work interrupted mid-task, context lost between sessions, and premature claims of completion. Source: Anthropic. None of those failures is fixed by a better model on its own; each is addressed in the surrounding system: progress written down as it happens, state kept outside the context window, tests as the definition of done, and restarts that resume from a record rather than from a summary the model has already forgotten.

Consistency over long tasks is still unsolved, and I do not expect the next model release to solve it. What the model cannot hold, the surrounding system has to carry. That is harness engineering, and it is the same requirement outside code: an assistant that forgets the ticket, or declares a refund processed because the request was phrased confidently, has a harness problem, not an intelligence problem.

The harness is the product

The clearest example is the one that started the current cycle. ChatGPT did not arrive as a new model so much as a new layer around one: a conversation, a set of instructions, and a boundary on what it could do. What people adopted was the harness.

That pattern is now the template for most AI products I see. A coding agent, a research tool, a support assistant, an internal automation: underneath each one is a model, and around it a harness that fixes what it can touch, what it remembers, when it stops, and what it is never allowed to do. The model supplies the capability. The harness supplies the behavior, and behavior is what a product is.

This is why the topic carries weight at the moment. Model releases arrive every few weeks and mostly raise the ceiling; the harness decides how much of that ceiling a given product actually reaches. An OpenAI team took that to its limit: five months, about a million lines, and 1,500 pull requests with no line written by hand, and the constraint that emerged was not generation but human QA capacity (OpenAI). At a smaller scale the consequence is the same: a team that can change the model behind a stable harness captures each improvement in an afternoon; a team that welded its workflow to one vendor’s specifics captures none of them, and discovers the cost of that decision when the vendor changes something.

Two shapes of this are worth separating.

Specialized harnesses are scoped to a single problem: code review, incident triage, extraction from contracts, analysis of a log format. Their advantage is not a better prompt. It is a smaller problem: the tools are the right ones, the acceptance criteria are known, the permission boundary is narrow, and the loop can verify against something the domain already defines. A general agent has to discover all of that at runtime, every time, with whatever context it can afford.

Composable harnesses are how that scales to a complex solution: several harnesses with contracts between them. Extraction, validation, decision, action, each with its own tools, model, verification, and budget, wired through interfaces that say what goes in and what must come out. The engineering is in those interfaces, much like splitting a service: a stage that lies about its output is a failure the next stage cannot recover from. Composition is also what makes a cheap model viable in one stage and unnecessary in another.

The consequence for teams is direct. The harness is where domain knowledge is written down: what “done” means here, which systems may be touched, what has to be checked before a result is accepted. That makes it the part of an AI product a company can own, since the model itself is rented. It is also where risk accumulates: tool definitions, credentials, and permission rules are as much attack surface as the model, and a permissive tool is a vulnerability with good manners.

Which suggests a design rule I have started applying: treat the model as a replaceable component behind the harness, not as the foundation of the design. Then the choice of model becomes an operational decision that can be revisited when prices, quality, or hosting requirements change, instead of an architectural commitment made once and defended afterwards.

The role of post-training

If the harness decides what a product does, the model still sets what it can do, and post-training is where part of that is decided.

Models have progressed on several fronts. Alongside changes in architecture and pre-training, post-training shapes specific behaviors: following instructions, solving problems, using tools.

One possible technique is reinforcement learning. In simplified terms, the model attempts tasks, receives evaluation signals, and undergoes adjustments that favor certain behaviors. Methods vary across models and organizations.

In coding, tests can provide part of that signal. But passing the available tests does not guarantee that an implementation covers every requirement or is suitable for production, which is the same gap described earlier, now moved into training.

The DeepSeek-V3 technical report describes supervised fine-tuning and reinforcement learning stages. Xiaomi also maintains a public page with MiMo RL metrics.

Data, architecture, training, inference infrastructure, and harness all contribute to the result. What matters when choosing a model is narrower than that: how it behaves once the harness around it is yours. A model that only performs well inside its vendor’s product, with that vendor’s tools and prompt shapes, is a good product; it is not evidence of a good model for the work you are building.

That makes portability a behavior rather than a checklist item. Chat completions and a tool-call schema are the easy part. The harder question is whether the model still follows instructions, calls the right tool, and stops when it should, when the instructions, the tool set, and the stopping rule are ones you wrote. A leaderboard does not run that test; your own harness does, and it is the only one that decides.

Knowledge is not the same as usefulness. GPT-3 holds a great deal of it, enough to answer questions that would impress a reader, and it cannot use a tool, follow a convention, or hold to a loop, because none of that was in its training. A model that reads as less intelligent in conversation, but runs the loop, calls the right tool, and respects the rules the system set, delivers more than a more knowledgeable one that only answers. A person judging intelligence and a harness judging completed work are measuring different things, and the second is what a product is built on.

There is a reason to expect a current model to pass that test. Post-training teaches tool use and instruction following against concrete formats, so providers train against their own call conventions and, increasingly, against shared ones: JSON-schema structured output, MCP, and the prompt shapes the popular agents publish. A model tuned on those conventions behaves more predictably in a harness that follows them, and less predictably in one that invents its own for no reason. Building a harness is therefore not free invention. Tool names and descriptions, error shapes, and stop conditions should resemble what the model was trained to see; the domain knowledge, the permission boundary, and the verification are where your design earns its keep.

The landscape, from daily use

What I see most of the time is coding tools, because that is the job: an editor with a model inside, an agent that opens a pull request, a review bot, a test generator. Within software development and automation, the options include families such as GPT, Claude, Gemini, Mistral, DeepSeek, GLM, and Kimi, and capability, price, tool integration, and hosting requirements carry different weights depending on the workload.

In daily work I find GPT and Claude the most convincing for development and agentic tasks. For the balance of capability and cost, the alternatives I reach for most are DeepSeek, GLM, and Kimi: in the automations I run the output is usually the same, the cost is often an order of magnitude lower, and the response comes back faster; only rarely do I need to bring in a larger, more expensive model. Gemini still falls short of my expectations in these uses. That is an impression from the models, versions, and environments I have actually used, not a result, and not a comparison anyone else can reproduce from this paragraph alone.

What surprised me after a while in Cursor is how far the cheap pair goes. Grok with Composer handles most of my coding work, and I rarely reach for a larger model — Opus or Fable, Sol or Astra — to finish something. When I do reach for one, the limitation is usually mine: a vague requirement, missing context, or a symptom described instead of the behavior. Written properly, the task rarely needs the frontier. On Cursor’s plan that pair costs little enough that I no longer hesitate to leave an agent working on something that does not deserve a premium model.

My expectations for Google remain high because of what the company brings together. Researchers affiliated with it helped create the Transformer, the architecture at the center of today’s LLMs. Google also develops its own TPUs, operates cloud infrastructure at scale, and has distribution channels such as Android and Search. It publishes its own coding and agentic results and ships developer integrations such as the Gemini API Docs MCP. My criticism is that, in these workflows, those resources have not consistently become a better experience, and I am careful to note that the perception alone does not prove that the company’s internal processes explain it. Gemini 4 is shipping now, and we will see what Google brings to the table with it.

Coding is the baseline; behavior is what separates them

Coding is the baseline, and most models have cleared it. A model that leads no benchmark still writes code that passes review: it follows the project’s conventions, handles the error paths, and takes a correction without needing it repeated. Judging these systems by the code they produce says very little, because acceptable code is now common.

What separates them is behavior. Whether the model works with the systems a team already runs, not only the ones inside its own product. Whether it keeps that behavior under a harness somebody else wrote. How long it operates without a person stepping in, and whether it stops when it should instead of reporting success on work it did not finish. Whether it follows the rules and guidelines it was given when the shortcut is cheaper than the rule.

None of those four has a leaderboard. All four are answerable only by running the model on your work, with your tools, for long enough to see where it drifts.

Those qualities are also what move a model past the role of chat responder or code generator. Answering a prompt well is a knowledge test; solving a real problem requires working inside the systems where the problem lives, respecting the rules around them, and continuing until the work is finished rather than until the answer sounds complete. Without them, a model is a faster way to produce text. With them, it can take on a process.

The interesting question is not what these systems do in an editor. It is what a team does with a model and a well-built harness when the problem is not a repository but a process.

The value is in the process

Seen that way, a model plus a specialized harness is raw material: a way to turn a manual, expensive, poorly documented routine into software. Intake of documents in formats nobody agreed on. Reconciliation between systems that were never designed to talk to each other. Triage of alerts that a person reads at three in the morning. Summarizing a procedure nobody has had time to read end to end. Each of those has a known input, an observable result, and a cost that a spreadsheet can show.

None of this is a demo. It is the sort of work that used to require a project, a budget, and a quarter of meetings. What the customer receives is not a model; it is the disappearance of a step.

I think the product managers and executives who already understand this are ahead of the ones waiting for a clearer directive. What matters is which processes are expensive, repetitive, and verifiable enough to be worth turning into a product, and then having the discipline to finish that product instead of admiring the idea in a backlog. Testing an idea used to cost a quarter; today it can cost a week. What did not get cheaper is the judgment about which idea deserves that week, and which deserves the months after it.

Because the range of processes is wide, no benchmark answers the question for you. The same model can handle alert triage well and invoice reconciliation badly, and the difference lies in the inputs, the rules, and what counts as correct in each case, none of which a leaderboard measures. What I like about a model in daily use is not evidence that it will work in a finance workflow. What decides that is a set of acceptance criteria written for the workflow and run against the model on real cases.

From benchmarks to evals

What follows is about building an AI product: the model inside it, and — more important — the evals that decide whether that model works in it. It is not about choosing a model to write your code. They are different decisions. A coding assistant can be swapped next week and costs you a habit; a model inside a product sits behind your prompts, your tools, and your acceptance criteria, so replacing it is a change to the product. Benchmarks answer the first question far better than they answer the second.

Benchmarks are the filter. Artificial Analysis groups evaluations by category — agentic knowledge work, terminal use, physics (CritPt), plus domain indexes for finance, legal, healthcare, and engineering — and LiveBench runs long-horizon suites and reports cost per successful task, which is closer to the number I care about. Between them, I can put together a shortlist of three or four candidates for a given job in about an hour.

A shortlist is all they give, for three reasons. The score is an average over a distribution that is not yours: a model near the top of a physics benchmark can be worse than a cheaper one at pulling fields out of scanned invoices, because the benchmark measures reasoning over clean problems and the work is mostly dirty inputs, business rules, and a schema nobody documented. Scores also age: in 2026 OpenAI withdrew its recommendation of SWE-bench Verified over test issues and contamination, then found problems in a large share of SWE-bench Pro tasks and withdrew that one too. Sources: Verified analysis and Pro analysis. And the same input can take different paths on different runs, so one successful demonstration proves little. A failure rate you have not measured is a failure rate you will meet in production.

What decides is the eval: a set of tasks taken from the real workload, each with a written acceptance criterion, run against every candidate model under the same harness. This is the part of building an AI product that teams skip, and it is the part that answers what the benchmark could not. An eval tells you whether the model covers what the product needs — which inputs it handles, which it gets wrong, and what the wrong ones look like — rather than how it ranks against other models on work nobody in your company does. It is also the only way to compare two candidates before shipping, because it holds the harness, the prompts, and the criteria fixed and changes one variable. The output is a distribution, not a verdict: how often a result is acceptable, and how bad the tail is.

Once the eval exists, it stops being a selection tool and becomes the test suite for the product’s behavior. That matters more than model choice, because the behavior depends on a surface that changes without anyone deciding to change the product: a prompt edited to fix one case, a tool added, a tool description rewritten, a field renamed in a schema, a provider shipping a new model version under the same name. Any of those can move results on cases that were working, and none of them appears in a benchmark. An eval set run on every change turns an invisible regression into a failing case before a customer finds it. Without one, every prompt edit is a release with no test.

With that suite in place, the operational decisions get easier. Among the models that clear the bar, it says which is cheapest per accepted result. And when the primary model rate-limits, fails, or is deprecated, it says which other model still passes, and how much the output changes when it takes over. Record versions, tools, and permissions with each run, along with acceptance rate, latency, retries, cost, and review effort; the numbers stay comparable across releases because the tasks do not change when the models do.

None of it means testing exhaustively before release. Coverage costs time, and proving that every branch behaves is time a competitor spends shipping. What matters is knowing which failures you can afford to meet in production and which you cannot, then building the monitoring and rollback that make the first category survivable. If budget is not a constraint, using the most capable model for everything does ship; it is also how a product ends up paying for capability nobody measured.

Cost is the next question, and it is where most model comparisons go wrong.

What a task actually costs

Every team that takes this past a prototype eventually asks what it costs to operate. The number that matters is not the one on the price list.

The unit is the completed task

Among the solutions that already meet the quality bar, cost decides what is viable at scale. A solution needs to balance quality, latency, throughput, availability, and cost, and in many scenarios a model that is good enough and inexpensive delivers more value than a frontier model used for every call. Most workflows contain far more cheap calls than hard ones.

Comparing price per million input tokens is not enough. The bill also depends on:

• Output token and cached input prices.
• The amount of context sent with each call.
• The number of calls, tool calls, and retries per task.
• Error frequency and the need for reprocessing.
• Human review time.
• Latency, availability, privacy, and hosting requirements.

The unit of comparison is cost per task completed at an acceptable quality level. Failed attempts belong in the denominator, not in a footnote. A cheaper model that needs more retries, tool calls, or context can still win by a wide margin, and a cheap model that produces work requiring senior review can lose even when the invoice looks small.

In my personal scheduled automations on Hermes, for example, I use LLMs to process complex, non-standardized data. With Claude Sonnet 5.5, the cost typically ranged from $15 to $20 per day. After switching to DeepSeek-V4.1-Flash, it dropped to less than $1 per day, with no noticeable loss of accuracy on those tasks.

That is one workload and an impression of quality, not a controlled experiment. Daily spend alone does not establish cost per task or equivalence between the models; reproducing it would require the observation period, the execution volume, the acceptance criteria, and the cost of every stage around the model. I mention it because the difference in magnitude is the argument: at that ratio, the question stops being which model is better in general and becomes which model is good enough here.

One lesson arrived late and cost me time: do not optimize cost before there is a user. A projection built from token prices and guessed volumes is usually wrong in both directions, so the hours spent trimming a bill that does not exist yet are hours not spent learning whether anyone wants the product. Put it in front of real use, measure what it costs per task, and optimize after the value is visible. Premature optimization is the old warning in software, and it applies to cost as much as to speed: before there is a user, the number is a guess, and refining a guess does not improve the product.

Where the savings come from

The options for lowering that number are not only commercial. The DeepSeek-V3 report described Mixture of Experts and Multi-head Latent Attention as ways to make training and inference cheaper. Source. The generations after it went further. DeepSeek-V4.1-Flash, released in September 2026 and since revised, is a 552-billion-parameter mixture of experts on a new causal encoder–decoder architecture, with 8 billion parameters active on input and 16 billion on output, and a KV cache that needs a quarter of the HBM and an eighth of the SSD storage of the previous generation. Source: DeepSeek.

Those changes show up in the price. Served as deepseek-flash, V4.1-Flash costs $0.15 per million input tokens and $0.60 per million output tokens at off-peak rates, which are half the peak rates, with cached input at $0.003. Source: DeepSeek pricing. More telling than the absolute number is the comparison inside one vendor: DeepSeek reports that V4.1-Flash beats V4-Pro on quality, cost, and runtime, and is retiring V4-Pro instead of continuing to sell the more expensive model. Z.ai’s GLM-5.3-Flash sits in the same band, at $0.15 input and $0.50 output against $1.40 and $4.40 for GLM-5.3 in the same family. Source: Z.AI pricing.

Competition among providers, including Chinese companies, has expanded the choices for balancing capability and cost, which matters most in high-volume work such as continuous analysis of logs, traces, metrics, and documents. Open weights push that further: because DeepSeek publishes the model, the price is not set by one vendor. On OpenRouter, dozens of providers serve the same weights at prices that undercut the first-party API and move with demand, so the buyer’s question stops being only which model to use and becomes which host to route to, checked against what each one actually delivers. It is no longer a Chinese-only story either: OpenAI’s GPT-6 Luna, at $0.10 per million input tokens and $0.50 per million output, sits at or below every figure above, and in the agentic work I run it delivers results I would not have expected at that price. Source: OpenAI pricing. It is the only low-cost model from a Western vendor I currently put in the same comparison.

Price is not the only lever, and not always the strongest one. Beyond choosing a cheaper model, the same workload costs less when repeated context is cached, when a step is replaced by explicit rules or plain code, and when a workflow is split so a small model handles what a large one was paying for. Those changes often save more than a discount does, because they remove work rather than discount it. That is the next section.

Cost is a dimension, not the thesis. It narrows the field after quality, and it says nothing about what may act without supervision. A cheap solution with an error you cannot afford is not a solution.

Choosing the technique to fit the task

A good part of the engineering value in AI work is deciding how much model a step deserves. The rest is deciding what must be checked.

Capability proportionate to the problem

Small language models (SLMs) are candidates for tasks with a defined scope and a verifiable result. Consider an automation that receives data in different formats, extracts relevant fields, and passes structured output to another system. Although the workflow processes a large volume, each run may need only a limited set of transformations. Generating boilerplate, adapting code to an existing convention, or summarizing logs can also be a starting point for evaluating a smaller model.

Risk sets how rigorous that evaluation must be. An incorrectly interpreted field can affect a business decision, even in a seemingly simple transformation. The relevant cases need to be checked, and the workflow needs a defined response to incorrect or incomplete output.

Code, rules, and specialized models

An LLM can accelerate the first implementation while we are still learning about the data and its variations. As patterns and requirements become clear, part of the processing may be better replaced by code, explicit rules, regular expressions, or parsers. For classification and prediction, specialized machine learning algorithms are also an alternative.

Another option for classification and routing is a System 1 decision model. Unlike an LLM used to generate text, models such as Jev, available on OpenRouter, take an input and return a structured decision, which option applies and with what confidence, instead of prose. They are still statistical models: structured output does not guarantee a correct decision.

The gain needs to justify implementation, maintenance, and infrastructure costs. For a workflow that runs a few times, migration may never pay off; at high volume, recurring savings justify the investment. As data and patterns change, monitoring errors and revisiting rules or models becomes part of that cost.

Escalating what needs more capability

Take the invoice pipeline from earlier. A parser reads the fields a parser can read: supplier, date, line items, total. Whatever it cannot read goes to a small model together with the schema. Both outputs pass the same checks: the total has to equal the sum of the lines to the cent, the supplier identifier has to match the record, and the date has to fall inside the batch being processed. Anything that fails a check goes to a larger model with the document and the reason it failed. Anything that fails there goes to a person with a one-line explanation. Most documents never leave the parser, and most of the rest never leave the small model.

The pattern has a name — routing — and Anthropic describes it as sending the common question to a smaller model and the unusual one to a larger one (Anthropic). In the pipeline above, the routing is not a judgment about how hard a document looks. It is three checks you can write, plus a validator model that re-reads the document and compares its answer. If a step cannot produce a check like that, you are not routing, you are guessing, and the guess will look fine on the cases you happen to see.

Whether it pays depends on how many documents each step settles. If the parser handles most of them and only a handful reach the large model, the average cost per document stays close to the parser’s cost; if half reach it, the cheap steps were decoration. That share is measurable on a sample before you build anything. The harder question is whether the escalations were the right ones: run the eval with escalation turned off and compare. If most of the escalated documents come out correct at the cheap step anyway, the pipeline is paying twice for the same result.

Several agents, arranged as a graph

Another option is to arrange the work as a graph: one node per stage or role, edges that decide what runs next, and each node with its own context, tools, model, and budget. Subagents are the same shape under one coordinator. It does not replace the techniques above, it arranges them, and a node can be a small model, a rule, or an escalation step.

The trade is context against coordination. Each agent sees less context and fewer tools, which gives the model less to get wrong; independent stages run in parallel; each stage can use the model it deserves; and a failure shows up at the step that broke instead of inside a long conversation that went nowhere. Against that, every edge loses meaning, because the next agent receives a summary rather than the reasoning behind it, and every handoff adds tokens, latency, and retries. Coordination brings loops and stalled waits that no single-agent test covers, and the eval grows faster than the number of stages: the eval of five stages is not the sum of five evals. It is worth it when the work actually separates into stages, or when verification needs a second party that does not share the first one’s instructions, as with a writer and a reviewer. When one agent with good tools already works, the graph adds coordination and removes nothing.

The question starts with the workload, its risks, and the required success rate: which combination of code, models, and validation solves this task at an acceptable quality and cost?

Autonomy requires proportionate limits

After the workflow, its quality, and its cost comes the decision that cannot be measured: what the agent may do without asking. Tools set the reach of an error. An agent reading a copy of the logs can leak them; an agent holding production credentials and permission to run commands can drop a table. The difference between the two is a permission, not a better model.

Two things set the line: how much damage a wrong action does, and whether that damage can be undone. Impact comes first, because the critical actions are the ones where no undo exists — a payment, a sent email, a signed contract, a deletion that has no backup. A leaked copy of the logs and a refund paid to the wrong account are not the same mistake, whatever the permissions say. Reversibility then decides how the gate works: where an undo is real, a checkpoint and a recovery path are enough; where it is not, the confirmation has to come before the action, every time. Batch work adds a third case: an agent that stops at 40% of its items leaves a state someone has to reconcile, so it also has to report which items it finished.

Where an action is critical, the confirmation cannot be a request the model may skip. Asking it to check before refunding is a prompt, and a prompt is advice: the model can decide the case is clear and proceed. The gate has to sit outside the model, in the code that executes the action, where the tool refuses to run without an approval token the agent cannot produce on its own. Then human in the loop is a property of the system instead of a behavior you hope for, and it still holds on the day the model is confident and wrong. Böckeler gives the difference a name: a request to the model is an inferential signal, expensive and non-deterministic; a token checked in the code that executes the action is computational (Böckeler). A gate built out of a prompt is the first kind, presented as the second.

That also defines what the person receives: the action, its arguments, the evidence the agent used, and a way to reject it with a reason. A dialog with an approve button and no context trains people to click approve, which turns the gate into a record of consent rather than a check.

For the customer, none of this appears as a permission model. It appears as what the tool does on its own, what it asks about first, and how it behaves after it gets one wrong. Nobody reads the matrix; the first unsupervised mistake is what they remember.

If the agent reads untrusted content — pages, documents, issues, messages — some of that content will carry instructions. That is prompt injection, and it cannot be handled by asking the model to ignore them, because the model is what is being attacked. What bounds the damage is what the agent can do once it is misled: which credentials it holds, which tools it can call, and which of those calls need a person. Source: Anthropic.

The practices that follow are few:

• Grant the permission the task needs, not every permission the tool set makes convenient.
• Keep the credentials for acting out of the step that reads untrusted input.
• Separate reads from writes, and let the agent have one of them.
• Require confirmation where the action cannot be undone.
• Record the tool, its arguments, and the result, so a failure can be reconstructed instead of argued about.

Permissions say what the agent may call. What surrounds the call decides how much a mistake costs: validated arguments, a sandbox for code, an allowlist for destinations, limits on money, and a rollback where one exists — chosen per tool, and strictest for the actions that cannot be undone.

What the extra capacity is for

Suitable tools, validation, and limits on autonomy expand a team’s capacity to investigate and implement. What that capacity produces depends on where it is applied.

In most companies, ideas for products and features have always outnumbered the capacity to build them. With limited time and people, only some ideas considered strategic moved forward; the rest waited in a backlog or were never tried.

The ability to generate a prototype or proof of concept quickly, often filed under vibe coding, changes that picture. An idea can be made functional, tested, and adjusted in shorter cycles, so the team learns earlier what deserves investment and what does not solve the problem, before committing people to a full implementation.

A working prototype is not a production-ready product. Turning it into reliable software still requires architecture, code, tests, security, integration, operations, and maintenance. Someone has to define requirements, understand the data, review changes, and respond when the system fails. Validating more ideas also creates more work: the ones that prove valuable must then be built, integrated, and supported.

This is where mastery of the tools becomes an advantage, not as a replacement for engineering judgment, but as leverage on it. The opportunity is to shorten the distance between an idea and evidence of whether it is worth building, learn from that, and iterate. Exploration capacity grows; when exploration produces value, demand for engineering grows with it.

For that reason, I think companies carrying out mass layoffs with AI as the justification may be making a strategic mistake while product opportunities exist and the business can absorb a higher rate of delivery. Reducing the team removes knowledge and execution capacity precisely when it becomes cheaper to test more ideas. A competitor that keeps a strong team and applies AI to its work can try more things and reach the market first, provided it turns the extra productivity into software customers actually want. Productivity that never leaves the repository is not an advantage.

Closing

Progress in AI expands what a team can investigate, implement, and operate. It does not decide what is worth building, and it does not absorb the consequences when something fails in production. Choosing tools to fit the problem, deciding what has to be verified by a person, and granting autonomy in proportion to risk remain engineering decisions, and they still belong to people who understand the code, the data, and the operations.

From where I sit, the job has already moved. Being a developer is less about writing code than it was, and each release moves it further: the code is what a model produces fastest, and what remains for a person is the problem — which one, for whom, at what cost, and whether the result is worth shipping. That does not shrink the craft, it changes its center. Working this way demands a wider base than syntax: the tools and techniques available, the systems the software touches, and the domain the problem lives in, because a model can only be as useful as the person directing it. What is left to us is the part no model frames on its own — noticing an opportunity, deciding to pursue it, and combining what we know into something that did not exist. That is where human creativity stops being a slogan and becomes the difference between a feature and a product.

This article drifted, and I mixed ideas that each deserved their own piece. What is above is what came to mind as I wrote, in the order it came. Some of those subjects — evals, harnesses, cost per task, autonomy — will come back in later articles, with more room than they had here. I hope some of it is useful to whoever is reading, and I will see you in the next one.

References and further reading

What follows is not a bibliography: it is the base I used, and what each piece supports in this text.

The argument

• METR, Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity and the February 2026 update: the two numbers behind the thesis, and the admission that the instrument stopped measuring.
• Kasra Vaziri, The Verification Bottleneck: the product-team version of the same diagnosis, with METR and GitClear.
• Riya Sharma, Verification is the bottleneck: a concrete case of verification becoming a feature.

The harness

• Birgitta Böckeler, Harness engineering for coding agent users: the name of the mental model, and the computational/inferential distinction I use in the autonomy section.
• LangChain, The Anatomy of an Agent Harness: why the same model performs differently under different harnesses.
• OpenAI, Harness engineering: what happens to a team when the constraint becomes human QA capacity.
• Marmelab, The State of AI Harness Engineering 2026: the one number in the round that isolates the harness from the model.

The limits

• Anthropic, Building effective agents, Effective harnesses for long-running agents and Trustworthy agents: what is already documented about routing, state, and autonomy, so I do not present it as new.