Two of Six AI Models Wrote a Shop That Sells More Than the Warehouse Holds

I gave six language models the same task: write a shop selling a limited edition vinyl, exactly one hundred units, not one more. Then I ran every solution and sent 400 requests into it, 60 in parallel.

One model sold 133 units. Another sold 125 and destroyed its own order file along the way. The remaining four held exactly one hundred.

Below: what the task looked like, what those two models broke, and why one of them looks correctly written.

The task

The prompt was short and deliberately contained no mention of concurrency, races or locks. I grepped for those words before sending it, because one of them would have turned the test into a check of whether a model can follow an instruction.

[ INSERT CODE BLOCK #1 — LANGUAGE: PLAIN TEXT ]

The information the risk follows from sits in the second sentence: the sale is announced in a newsletter for a specific hour. That means several hundred people click within the same second. I never said so directly, and that was the point.

Every solution ran from a clean state in two configurations: one uvicorn process and four. The second is the production standard.

deepseekv4pro: nothing guards the write

The code does four things in order: load state from the file, check whether anything is left, decrement the counter, write the file. Nothing guards the gap between the read and the write.

With four processes, two requests read the same number, both see a free unit, both sell it. Result: 133 units out of one hundred. Thirty-three people need a refund.

More interesting is what happens with one process: 702 parse errors and 351 responses with status 500. Worse than with four. The reason is easy to miss: the handler is a plain def, not async def, so FastAPI runs it in a thread pool. Threads inside one process trample the file exactly like separate processes do.

There is no configuration in which this code works correctly.

kimi3: looks correct and works most of the time

This case is more interesting, because at first glance everything is in place. There is a threading.Lock. There is with lock spanning the whole operation. There is a write through a temporary file and os.replace, the textbook atomic write. The README states that "reserving a unit is atomic, so a race condition will not sell the same unit to two people".

With one process: one hundred sold, zero errors. Everything checks out.

With four: 782 parse errors, 394 responses with status 500, and in an independent measurement 125 buyers holding an order number, including the same number handed to two different people. The file keeps one hundred records, so twenty-five of those sales exist only in customers' inboxes.

Two things failed at once.

threading.Lock applies within a single process. Four workers means four independent locks that know nothing about each other.

The second cause is subtler, and it is the one that destroys the file. The temporary filename is fixed:

[ INSERT CODE BLOCK #2 — LANGUAGE: PYTHON ]

os.replace is atomic, but only when the source file belongs exclusively to you. Four processes write to the same state.json.tmp, so the rename publishes a file spliced from two halves. From that point the state is unreadable and every subsequent request ends in an error, until someone fixes it by hand.

The whole difference between "works flawlessly" and "falls apart and oversells" sits in one startup flag.

What this means after deployment

The newsletter goes out at noon. The wave arrives in the first minute.

With deepseek's code, thirty-three people receive a purchase confirmation for goods that do not exist. That means refunds, support tickets and public reviews.

With kimi3's code it is worse, because data loss joins in. Some confirmed orders vanish from the file on the next write, and shortly after the application stops responding entirely. You have customers holding proof of purchase who are not in your system, and you do not even know who they are.

Worth adding: with one process kimi3 behaves impeccably. On a developer machine, in integration tests and on a single-worker staging environment, this code looks correct.

Two takeaways

A single-process test does not test production

Had I only run the default configuration, kimi3 would have scored high. The difference between a correct and a catastrophic result is the --workers flag. If your integration suite starts one process, you are not checking what reaches the server.

Verifying it is cheap. One loop is enough:

[ INSERT CODE BLOCK #3 — LANGUAGE: BASH ]

A README is a claim, not a measurement

Kimi3 documented a safeguard that does not work beyond one process, and never noted that limitation. The documentation shows a run with --reload and nothing else, so it is the doc itself that pushes an operator toward scaling out before heavy traffic.

This is not about bad faith on the model's part. It is that a description of code never substitutes for running it.

What this test does not measure

This is one run per model. Temperature and sampling randomness affect the result, so I treat it as a probe rather than a metric.

The test also only checks the reflex when the requirement is stated explicitly. The prompt said "exactly one hundred units", so the models knew what to guard. Four of six delivered it, which is a better result than I expected.

The natural next question is whether swapping the file for a real database would change anything here. I measured that across ten storage variants, and the result surprised me enough to deserve its own piece.