Two of Six AI Models Sold More Than the Warehouse Had. A Database Would Not Have Fixed It

Two of six language models wrote a shop that sold 133 units against a stock of one hundred. The obvious fix suggests itself: use a real database instead of a file.

So I checked. I took the same naive logic — read the order count, check it is below one hundred, append an order — and ran it through ten variants of the storage layer, 400 concurrent purchase attempts and three runs per variant.

The database did not help. On SQLite that code sold 139 units out of one hundred, on Postgres 122–127, in both cases without a single error returned to a client. Raising the isolation level to REPEATABLE READ changed nothing.

Four variants landed on exactly one hundred: the file with a lock, SQLite with BEGIN IMMEDIATE, Postgres with SELECT … FOR UPDATE, and Postgres with the condition written into the update itself.

Below: what those two models broke, why REPEATABLE READ does not help, and why the database version is in practice more dangerous than the file version.

Where this came from

I am building a set of coding tests for language models. Test four reads like this: we are selling a limited edition vinyl, exactly one hundred units, not one more. The sale is announced in a newsletter for a specific hour. One person, one unit. No database, state in a JSON file.

The prompt contains no words like "concurrency", "race condition" or "lock". What it does contain is a start time announced in a newsletter, from which it follows that several hundred people will click within the same second.

Six models, one identical task. Every solution ran from a clean state, at one and at four uvicorn workers, with 400 requests at 60 in parallel.

Four out of six held the pool. That is a qualitatively different result from the previous test, where I asked about something the prompt stayed silent on and seven out of seven failed. When the requirement is written out explicitly, most models deliver it.

The two that did not fail in ways worth describing.

deepseekv4pro: no lock, in any configuration

The code reads: load state from file, check whether anything is left, decrement the counter, write the file. Nothing guards the gap between read and write.

At four workers it sold 133 units out of one hundred. Thirty-three people need a refund and an explanation.

More interesting is what happens at one worker: 702 parse errors and 351 responses with status 500. Worse than at four. The reason is simple and easy to miss: the handler is a plain def, not async def, so FastAPI runs it in a thread pool. Threads inside one process trample the file exactly like separate processes do. There is no configuration in which this code works.

kimi3: correct on one process, catastrophic on four

This case is more interesting, because at first glance everything is in place. There is a threading.Lock, there is with lock around the whole operation, there is a write through a temporary file and os.replace, the textbook atomic write. The README states that "reserving a unit is atomic, so a race condition will not sell the same unit to two people".

At one worker: one hundred sold, zero errors. Everything checks out.

At four: 782 parse errors, 394 responses with status 500, and in an independent audit measurement 125 buyers holding an order number for a hundred-unit edition, including the same number handed to two different people.

Two things failed at once and both are instructive.

threading.Lock applies within a process. Four workers means four independent locks that know nothing about each other.

The second cause is subtler. The temporary file has a fixed name:

[ INSERT CODE BLOCK #1 — LANGUAGE: PYTHON ]

os.replace is atomic, but only when the source file belongs exclusively to you. Four processes write to the same state.json.tmp, so the rename publishes a file spliced from two halves. Atomicity is formally preserved and means nothing in practice. From that point the state is unreadable and every subsequent request is an error, until someone fixes it by hand.

Worth noting: fable5 has exactly the same construction with a fixed temporary filename. The only thing saving it is that the write sits inside a filelock, which works across processes. Remove that lock during a refactor and you have a second kimi3.

The question that asked itself

If the file is the problem, what would a real database do? A database has transactions, row locks, isolation levels. It looks like the problem disappears on its own.

I measured instead of guessing. I wrote the same application logic across ten storage variants, changing only how the data persists:

[ INSERT CODE BLOCK #2 — LANGUAGE: PYTHON ]

400 attempts, 40 threads in parallel, limit one hundred. Three independent runs to rule out coincidence. The script is in the repository and reproduces with one command.

The results are in the table at the top. Four things follow from them.

1. I had to retract my own claim about SQLite

Before running the experiment I said something like this: SQLite will accidentally save naive code, because it has a single writer and serializes writes itself.

That is wrong. SQLite came out worse than Postgres: 139 units out of one hundred, repeatably across all three runs.

The mechanism: the SQLite lock serializes individual write operations, not the sequence "read, decide, write". Under default autocommit every SQL statement is its own transaction, so SELECT COUNT(*) finishes and releases the lock before the INSERT even starts. Two threads fit comfortably in that gap.

The same database with BEGIN IMMEDIATE lands on exactly one hundred. The difference is not the database, it is when the write lock is taken: upfront, or after the decision.

2. A higher isolation level is not enough

I expected REPEATABLE READ to settle it. It did not: 121–126 sold, practically the same as READ COMMITTED.

The reason is instructive. REPEATABLE READ protects against a write conflict on the same row. Here every buyer inserts their own new order row. Two transactions each see 99 orders, both append the hundredth, neither conflicts with the other, both commit. In database terminology that is a phantom rather than a lost update, and only SERIALIZABLE detects it.

The Postgres documentation says as much about READ COMMITTED: "This behavior makes Read Committed mode unsuitable for commands that involve complex search conditions". Worth reading before assuming the default isolation level guarantees anything.

3. The levels that do work have a price and a catch

SERIALIZABLE genuinely allowed no overselling. It also rejected roughly 375 of 400 transactions with a serialization error. Without a retry loop the sale ends at 25 units out of a hundred.

Same story with BEGIN DEFERRED in SQLite: 393–398 requests got SQLITE_BUSY, and 2–7 units sold.

Both configurations look safe in the table, because the oversell column reads zero. In practice they mean the sale did not happen and your warehouse is still full. That is a trap for anyone who only checks whether they sold too much.

4. Exactly one thing works

Four variants landed on exactly one hundred: the file with a lock, SQLite with BEGIN IMMEDIATE, Postgres with SELECT … FOR UPDATE, and Postgres with the condition written into the update itself.

[ INSERT CODE BLOCK #3 — LANGUAGE: SQL ]

All of them do conceptually the same thing: they make the decision and the write one indivisible operation. This is not a difference between a file and a database. It is the difference between "I check, then I write" and "I check and write in one move".

A model that does not grasp that difference with a file will not grasp it with Postgres. The only thing that changes is whether you get to see its mistake.

The most important number in the experiment

Put two rows side by side:

The file version shouts. Hundreds of errors, the file stops parsing, impossible to miss in any test environment. One run on a developer machine and you know something is wrong.

The database version stays quiet. One hundred and thirty-nine people get a green response and an order number. No errors in the logs, clean monitoring, green dashboards. You find out during a stock count, or from thirty-nine customer complaints.

From a business standpoint the second situation is worse, even though it looks better. That, in my view, is the main reason this class of bug survives code review and reaches production.

What the audits found beyond my measurements

I ran a separate security audit on two of the solutions. It returned things my test never checked, because I only tested the purchase endpoint.

In opus5, the solution that won the test, the admin panel is open by default. The protection exists, but it only activates once an environment variable is set. On a standard run, curl /api/admin/orders with no headers returns the e-mail address of every buyer. The same gap opens the "resend all failed mails" trigger.

The second finding is more interesting, because it concerns logic rather than code. There is no payment and no address verification, and reservations never expire. A simple script takes the entire run in 1.11 seconds using made-up addresses. The vinyl does not get oversold, it gets locked up, and real customers see "sold out" within the first second. From a business standpoint that is a worse outcome than overselling, and my test did not measure it at all.

The audit also corrected one of my measurements: at one worker kimi3 sells exactly one hundred, not the 97 I recorded. My number was a client-side artifact. That correction makes the solution more dangerous, not less: on a development machine it looks flawless.

What to do about it

Find your check-then-act

Everywhere the code reads state, decides in application language and only then writes, you have the same bug as kimi3 and deepseek. Limited stock, seat reservations, the last item in a cart, granting a discount code, anything with a bounded count.

Do not rely on the isolation level

READ COMMITTED does not protect you, REPEATABLE READ does not either when every caller inserts its own row. SERIALIZABLE does, but it requires retry handling, without which it converts an overselling problem into a not-selling problem.

Test with multiple processes

Kimi3 at one worker looks flawless and would have scored high. The whole difference between "works" and "catastrophe" sits in the --workers flag. If your integration test starts one process, you are not testing what goes to production.

Count records, not response codes

With a database, overselling produces not one error. The only way to see it is to count rows and compare against the limit.

Next round

The natural follow-up: the same task, but with Postgres instead of a file, and the question of whether the models that forgot the lock will reach for FOR UPDATE or a condition inside the UPDATE. My guess is no, because the problem is not API knowledge but the inability to picture two requests in the same millisecond. If that guess holds, it will be a stronger result than anything I have now.

The second question I want to measure: how much instruction is enough. Here the requirement "exactly one hundred units" was spelled out and four models delivered it. In the previous test, where authorization was never mentioned, nobody did. Where is the line?

The experiment script reproduces with one command and the test prompt is a dozen or so lines. If anyone runs it against other models, or another database, I would be glad to compare results.