The Prototype-to-Production Wall
Most AI prototypes never make it to production. The teams that cross the wall stop renting the model's guess and turn it into deterministic, owned capability. Semantic to mechanical: the pattern, the data behind the failure rate, and how to cross it.
We built an internal dashboard for tracking team objectives. Someone described what they wanted, an assistant wired it up against our data, and in the demo it was genuinely impressive. Ask it how a team was tracking against its goals and it answered in plain language, with charts, in seconds. Everyone in the room wanted it.
Then people started using it. A lead would open the dashboard, dig into their own team's performance, and get one assessment. A teammate would ask the same question about the same data an hour later and get a different one. The numbers behind the scenes had not changed. The tool had. Suddenly everyone was looking at their own private version of the truth, and a dashboard whose entire job was to give us a shared view of reality had quietly become an engine for disagreement.
That gap between the demo that wins the room and the tool the team can actually depend on has a name in our shop. We call it the prototype-to-production wall. It is the single most expensive thing about building with AI right now, and almost nobody talks about it until they are already stuck against it.
Two words: semantic and mechanical
A large language model is a semantic engine. You give it meaning and it gives you a plausible response. That is a superpower for a prototype: you can go from an idea to something that looks and feels real in an afternoon, without writing the rules down first. The model infers the rules from your intent, every single time you ask.
A business runs on mechanical systems. A payroll calculation, an eligibility check, an objectives rollup - these have to produce the same answer for the same input, today and next quarter, no matter who asks. That is not a nice-to-have. It is the definition of a system people can trust their decisions to.
The prototype-to-production wall is the seam between those two worlds. A prototype is semantic: it re-derives its behavior from meaning on every run. A production system is mechanical: its behavior is fixed, tested, and owned. The wall is what you hit when you try to put a semantic artifact in a mechanical job.
Semantic (the prototype)
- Behavior inferred from intent on every run
- Same question can yield different answers
- Fast to build, hard to trust
- Burns tokens every time it operates
- You rent the model's guess
Mechanical (production)
- Behavior fixed, tested, and versioned
- Same input always yields the same output
- Slower to build, safe to depend on
- Spends tokens to improve, not to run
- You own the capability
Why the same question gives everyone a different answer
When our dashboard handed two people two different assessments of the same numbers, it was not broken. It was working exactly as a semantic engine works. The logic that turned raw data into an assessment did not live in tested code. It lived in the model's head, re-invented on each request. And a model does not run the same instructions twice - it produces the most plausible next token given everything in front of it.
People assume that setting the temperature to zero makes a model deterministic. It does not. Researchers at Thinking Machines Lab ran the same prompt through a standard inference stack one thousand times at temperature zero and got eighty different completions, the first of which diverged from the rest after just a hundred or so tokens. The nondeterminism comes from how requests get batched and how floating-point math is ordered on the hardware, not from a setting you can flip off.
The uncomfortable part
If the rule that produces a number lives inside the model instead of inside your code, you do not have a calculation. You have an opinion that changes with the weather. That is fine for a brainstorm. It is a liability for anything a person will make a decision on.
The wall is where most projects die
The dashboard was not an unlucky one-off. Failing to cross this wall is the normal outcome for enterprise AI, and the numbers are stark once you go looking for them.
95%
of enterprise generative-AI pilots deliver no measurable business impact; only about 5% reach production with real returns
MIT Project NANDA, 2025
30%+
of generative-AI projects predicted to be abandoned after proof of concept by end of 2025
Gartner, 2024
80
distinct outputs from 1,000 identical runs at temperature zero on a standard inference stack
Thinking Machines Lab, 2025
The MIT NANDA study is blunt: the divide is not about model quality or regulation. It is a learning gap between a demo and a system that holds up under real use. Gartner's analysts reached the same place from a different direction, first predicting widespread abandonment after proof of concept and later revising that estimate upward. Proof of concept is not the hard part anymore. Modern tools make the semantic prototype almost free. The hard part is everything after.
Here is the trap in slow motion. A team wires up a few MCP servers, vibe-codes against them, and produces something that demos beautifully. The applause creates pressure to ship. But the thing that demoed was semantic, and the job it is being asked to do is mechanical. So it stalls at the wall in one of a few predictable ways.
Inconsistent results
The same input produces different answers for different people, so nobody can trust the output enough to act on it.
No way to fix a bug
When the answer is wrong there is no rule to correct, only a prompt to nudge. The fix for one case breaks three others.
Cost that scales with use
Every operation is a fresh inference call. The more the tool succeeds, the more it costs to run, forever.
This is the same failure mode I have written about from other angles: vibe coding is a prototype strategy, not an engineering methodology, and reliability in production comes from the system around the model, not the model itself. The wall is what all of those posts are circling.
The harness is the fusion point
You do not cross the wall by picking a better model or writing a cleverer prompt. You cross it by changing what the model is for. A model is excellent at proposing behavior. It is unreliable at being the behavior. So you use it to author a mechanical system, then you run the mechanical system.
The thing that makes that safe is a governed harness: an environment that constrains what AI is allowed to produce, checks it against your standards, and turns an approved result into tested, versioned code that becomes part of the product. The harness is where semantic meets mechanical. It is where the subject-matter expert's intent gets fused with an industry's hard rules and comes out the other side as software you can depend on.
It captures intent
Domain experts describe what they need in their own language. The model turns that into a candidate specification and implementation.
It enforces standards
Specs, rules, review, and automated verification gate every change, so a non-engineer's contribution is held to the same bar as an engineer's.
It produces mechanical output
The result is deterministic, tested code that runs without a model in the loop - not a prompt that re-guesses each time.
It stays extendable
New requirements flow through the same harness, so the system grows without a rewrite and without breaking what already works.
This is governance, not paperwork. A rule the harness enforces on every change is worth more than a document nobody reads. That distinction, and why it matters more as AI writes more of your code, is the core of how we think about execution excellence.
The move that compounds: keep the requirements, discard the guesswork
Here is the reframe that changed how we build. When a prototype "fails," it has usually already delivered its most valuable output, and it is not the code. It is the requirements. The prototype is how you discovered, in the cheapest possible way, exactly what the system needs to do and how people actually want to use it. That knowledge is the asset. The disposable semantic implementation was just how you found it.
So we stopped trying to harden prototypes and started harvesting them. With the objectives dashboard, we took the requirements the prototype had surfaced - the exact questions people asked, the definitions they argued about, the views they reached for - and ran them through our harness. Out came a production, scalable system with the calculations pinned in tested code. Now the whole team uses it, the same input gives everyone the same answer, and when someone needs a new cut of the data we add it as a new requirement through the harness instead of rewriting the thing. The tool that was tearing our shared reality apart became the thing that holds it together.
Spend tokens to build, not to operate
In a semantic prototype every operation is an inference call, so cost grows with usage and never stops. In a mechanical system the tokens are spent once, up front, to author and extend the capability - and then it runs for the cost of running code. You are not chasing "cheaper AI." You are converting a recurring rental into an owned, deterministic asset you can build a moat on.
That is the whole thesis in one line: use AI to go from semantic to mechanical. Let the model do what it is uniquely good at - turning meaning into a first draft of behavior - and then lock that behavior into a system that does not need the model to stay correct. This is not about replacing the people who understand the domain. It is about giving those people a way to turn what they know into durable technology, which is exactly the difference between documenting intent and enforcing it.
What this looks like on Monday
You do not need to reorganize anything to start. You need to be honest about which side of the wall a given piece of work belongs on.
Name the wall out loud. When a demo lands, ask the next question immediately: does this need to give everyone the same answer? If yes, it is a mechanical job and the prototype is not the deliverable.
Treat the prototype as a requirements machine. Capture what it taught you - the questions, the definitions, the edge cases - and plan to throw the implementation away.
Pin the rules in code. Anything a person will make a decision on gets a tested, versioned implementation. The model can write it; it should not be it.
Route changes through governance. New requirements come in as specs the harness enforces, so the system extends without regressing.
And be clear about where the semantic side is still the right answer. Not everything needs to become mechanical. Open-ended exploration, drafting, summarizing, one-off analysis, and genuinely creative work are jobs where a plausible answer that varies is a feature, not a bug. The mistake is not using a semantic tool. The mistake is asking a semantic tool to do a mechanical job and calling the result production.
The teams pulling ahead with AI are not the ones with the best demos. They are the ones who learned to cross the wall - to treat the model as a way to author capability rather than a way to operate it, and to turn the meaning it generates into mechanical systems they own.
The prototype earns you the requirements. The harness turns the requirements into a system. Semantic to mechanical is not a slogan. It is the difference between a tool that impresses a room and a tool the whole team can trust.
Cross your own prototype-to-production wall
We spend a lot of our time turning semantic prototypes into mechanical systems - specs, governed harnesses, and automated verification that let domain experts build durable, deterministic tooling. If your team keeps hitting the wall between an impressive demo and something you can depend on, I'm happy to share what we've learned.