The Bug That Only Shows Up the Second Time a Job Runs

Every system that calls itself reliable has retries somewhere in it: a webhook sender that tries again after a timeout, a queue that redelivers a job if a worker doesn’t acknowledge it in time, a cron that reruns a failed step instead of giving up. Retries are good. They’re also how a payment gets charged twice, a shipment gets created twice, and a customer gets the same “your order shipped” email three times in a row.

The thing retries assume, quietly, is that running the same code twice is safe. Most code was never written with that assumption in mind, because most code was written to run once.

“At least once” was never a promise about your code

Queues and webhook senders almost never guarantee exactly-once delivery. Pub/Sub, SQS, and basically every webhook provider I’ve integrated with promise at least once: your handler will definitely run, and it might run more than once for the same logical event. That’s not a bug in the queue. It’s the deal you signed up for in exchange for not losing messages, and it’s usually the right trade.

What it means in practice is that a worker can be most of the way through processing a job when its visibility timeout expires, the queue assumes it died, and redelivers the same message to a second worker while the first one is still finishing. Or a webhook sender gets a slow 200 back (your server did finish, the response just took eleven seconds) and retries because from its side that looked like a timeout. Nothing went wrong on either end. The system behaved exactly as documented. Your handler just ran twice.

Where this actually bites

The failure mode is rarely “the whole operation ran twice and everyone noticed immediately.” It’s quieter than that.

A payment webhook that creates a shipment. The handler charges nothing itself (the processor already did that), but it does call a shipping provider’s API to create a label and a tracking number. Run it twice and you don’t get an error, you get two labels and two tracking numbers for one order, and a support ticket three days later asking why two boxes showed up.

A queue worker that increments a counter. A “mark this video as watched” job that increments a view count is the kind of thing nobody bothers to make idempotent, because what’s the worst case, a view count is off by one? Except redeliveries aren’t rare under load, and off-by-one compounds into numbers that don’t match what the billing system calculated from the same raw events, which is a much worse afternoon than the original bug would have been.

A batch step that reruns from the top. A cron-driven import job fails on row 4,000 of 10,000 and the whole thing reruns on retry, including the first 3,999 rows that already succeeded. If row 1 was “insert a customer,” you now have two customers.

None of these needed a bug in the retry logic. The retry logic worked perfectly. The code it was retrying just wasn’t built to tolerate that.

The fix: check before you act, not after

The actual fix is almost always the same shape: give every logical operation a key that identifies it uniquely (a webhook’s event ID, a job’s idempotency key, whatever the sender already provides or you generate yourself), and record that key as processed before you do anything the outside world will notice, not after.

In MySQL that’s usually a table with a unique constraint, nothing cleverer:

INSERT INTO processed_events (event_id, created_at)
VALUES (:event_id, NOW())

Let the unique key do the work. If the insert fails with a duplicate-key error, you already handled this event, so stop here and return the previous result instead of repeating the side effect. Don’t SELECT first to check and then INSERT if it’s missing. Two workers can both run that SELECT before either one finishes the INSERT, both see “not processed yet,” and you’re back to running the side effect twice, just with extra steps and a false sense of safety.

The time the check came after the email

The version of this I actually got bitten by wasn’t the database part. The dedup check was correct. The order of operations around it wasn’t.

The code looked roughly like: send the “your invoice is ready” email, then record the event as processed. Reasonable sequence to write, wrong sequence to run, because if the process died or got redelivered in the gap between those two lines, the event was never marked processed and the next attempt sent the email again. The email had already gone out; the one thing that was supposed to prevent a repeat happened one line too late to catch it.

The fix was boring once it was visible: record the event as processed first, inside the same transaction, before touching anything external. If the side effect itself then fails, that’s a real failure and it’s fine for it to retry. What’s not fine is a side effect succeeding and the bookkeeping that was supposed to prevent a repeat arriving after it instead of before it.

Two assumptions that don’t hold

“Our queue guarantees exactly-once” is something people say about systems that, read closely, promise no such thing. Check the actual docs for whatever you’re using. Most of what gets called exactly-once in casual conversation is at-least-once delivery plus idempotent handling built on top, and the second half of that sentence is the part a team has to actually write.

“Retries are safe because HTTP is stateless” confuses two different things. HTTP being stateless means the server doesn’t need a session to understand the request. It says nothing about whether charging a card, creating a row, or sending an email is safe to do twice. Statelessness is a property of the protocol. Idempotency is a property of what your handler does, and you have to build that part yourself.

What idempotency actually buys you

The honest way to put this: idempotency isn’t a resilience feature you add on top of retries. It’s the precondition that makes turning retries on safe in the first place. Without it, every retry is a bet that nothing succeeded before the failure, only after it, and on a long enough timeline that bet loses. The fix costs one unique constraint and a moment spent deciding what the operation’s key actually is. That’s cheap, next to finding out from a support ticket which parts of a transaction already happened.