The bug was that nothing could go wrong

Thirteen posts were queued to publish. Every one of them advertised an offer we had stopped running and linked to a domain we had replaced.

We found this about eight hours before the first was due to go out. The fix looked simple: open the admin, reject them, push the corrected versions. The admin has a Reject button on exactly that screen.

The button did nothing. Every route we tried returned the same error: only draft posts can be edited.

The guard that was doing its job

The scheduler had four guards, one each on edit, approve, reject and delete. All four said the same thing: the post must be in DRAFT.

That reads as careful. A post nobody has approved yet is safe to change; a post that has been signed off should not be quietly rewritten. It is the kind of rule you write on a good day, thinking about the person who edits a post after someone approved it.

Meanwhile a cron ran every five minutes, selected every post whose status was APPROVED and whose scheduled time had passed, and published it.

Put those two facts next to each other. The only status the publisher acts on is the one status no other operation will touch. Approval was not a checkpoint. It was a trapdoor: the post went in, and the only way out was onto the internet.

Nobody designs this

It is worth being clear that nobody sat down and decided approved posts should be uncorrectable. Each guard was reasonable in isolation. The publisher query was reasonable in isolation. The bug lived in the space between two correct decisions, which is where this class of bug always lives.

You cannot find it by reading either piece of code. You find it by asking a question neither author was asking: what happens when someone needs to undo this?

We only asked because we were the ones needing to undo it, at eight in the evening, with a deadline. That is the expensive way to discover an architectural gap, and it is the usual way.

What we actually did

We pulled the posts with a direct database write. Thirteen rows, moved out of the publish queue by hand, after checking in a transaction that exactly thirteen would move and nothing else would.

That is not a fix. It is an admission that the software could not do the thing the software was for. It also required production database access to correct a typo in a marketing post, which is its own problem.

Then we fixed the actual issue. The transitions now accept any status a post can still be pulled back from, and refuse only once it is publishing or published, which is the point at which it has genuinely left the building. A post can be corrected right up until it fires.

The second bug, which the first one hid

Widening those guards created a race we had not had before.

The publisher polls APPROVED and SCHEDULED. Once an admin could also edit and delete posts in those states, both were reaching for the same row. The status checks were advisory: each read the status, did some work, then wrote, with nothing stopping the status changing in between.

Three ways that goes wrong. Two callers both see a publishable post and both publish it. An admin's edit lands mid-publish and reverts the status while the publish is still running, then the publish overwrites the edit. A post gets deleted while the publisher is holding it, and the publisher's error handler saves its stale copy back, resurrecting the row you just deleted.

The fix is one statement instead of two. Rather than reading the status and then writing, you write conditionally: update the row only if its status is still what you expect, and check how many rows changed. Exactly one caller gets a row. Everyone else gets zero and backs off. The database arbitrates, because the database is the only thing that can.

What we would tell you to check

If you have a queue with a scheduler on one side and humans on the other, three questions are worth an hour of somebody's time.

Which statuses does your background job act on, and can a human touch a row in those statuses? If the answer is no, you have this bug.

When a status guard passes, how long until the write happens, and can the status change in between? If it can, the guard is a suggestion.

What is the recovery path for something already queued, and has anyone walked it? Not whether the button exists. Whether it works.

The last one is the one that catches this. A recovery path nobody has walked is a recovery path that does not exist, and you find out on the day you need it.

We build and run systems like this for clients across Nepal and Australia, including our own. If you want a second pair of eyes on a queue before it teaches you something at eight in the evening, get in touch.