The three worst defects I shipped this year all exited zero. No stack trace, no alert, no red line in any log. Every run reported success, and by its own definition every run was correct.
The Failures Nobody Pages You For
One of them wiped a computed field on thousands of properties, once an hour, for weeks. One cut two thirds of a person's resume out of a prompt and got a model to deny a degree that person actually holds. One read a table, stopped at 1,000 rows, and said it was finished.
I found all three by hand, late, while looking at something else.
That's the part worth writing down. Crashes are a solved problem. We have error trackers, alerts, on-call rotations, dashboards that go red at three in the morning. What none of that catches is code doing exactly what you told it to do while the result is wrong.
I run two products by myself. One is an engine that fills out job applications, and I pointed it at my own job search first, so it filed 1,183 applications for me, 52 waves in 18 days. The other is Foreclosure Radar, which works out what a house heading to a forced sale is actually worth before someone bids on it.
The expensive bugs are the ones that never throw.
Here's all three: what each one did, what it cost me, and the change that makes that shape of bug impossible instead of unlikely.
Bug One: The Hourly Job That Erased the Product
Foreclosure Radar reads county auction lists, then enriches every lot: what the property is worth, what shape it's in, which debts survive the gavel. Pennsylvania takes the most work. Counties there freeze their assessment base year for decades, and one is still valuing houses on a 1961 base year, so the site computes the real number itself.
Importers and enrichers both write to the same row. Every enricher writes the whole row back:
await upsertLot(db, { ...lot, fullMarketValue })Read the row, change one field, write all of it. That's fine on its own.
The other side is where it went wrong. The Pennsylvania importers built their row from the county's catalog page, and for the fields a catalog doesn't carry they wrote an explicit null:
{ fullMarketValue: null, assessedValue: null, taxMap: null }Also fine on its own. Put the two together and you have a delete.
What actually happened
An hourly job re-imports every sale that's still open. So every hour the importer overwrote the computed market value with null, and every hour the enricher had to earn it back, and in between the site served nothing to anyone looking. Bucks 232 lots, Philadelphia 380, Schuylkill 93, Montgomery 90, Berks 61, plus all 110 Delaware parcel ids.
Nothing failed. The importer wrote what I told it to write. The database accepted it, because null is a legal value in a nullable column. The job logged the same success line it logs every hour of every day.
The thing that finally gave it away was a mismatch between a document and a screen. My own handoff doc said market value coverage was 65 percent. I opened the live site and almost nothing had one. Both numbers were true, at different points in the same hour.
The fix was five lines of types
Not a guard, and not a note in the code review template. A type:
type OptionalOnWrite = 'fullMarketValue' | 'assessedValue' | 'taxMap'Any column an enricher fills goes in that union, and for those columns the write type accepts a real value or nothing at all. It does not accept null. The old importer code stopped compiling the second I saved the file.
Then a regression test, named after the failure instead of after the function:
a re-import that omits parcel and value does not erase them
A day later the same class of bug showed up on two more fields, description and lot size, through a completely different script. The type caught that one in my editor before it ever ran.
Why "Impossible" Beats "Careful"
The same lesson landed from the other product, in a nastier way.
Dispatch on a vendor used to be an if-chain with one vendor sitting in the else branch. Adding a new vendor compiled cleanly and quietly routed it into the wrong adapter. In a system that fills forms with a real person's details and clicks submit, that is not a rendering glitch. Every dispatch is now a switch closed with assertNever, so a new member of the union breaks the build in every place that has to care about it.
Both fixes are the same move, and it isn't about being disciplined. It's about deleting the state where discipline is required.
| Approach | What it costs | What it catches |
|---|---|---|
| A note in the code review checklist | Free, forgotten in a month | Nothing, once you stop remembering |
| A runtime guard | A branch and a test | The one case you thought of |
| A type that refuses to compile | Ten minutes, once | Every future case, in the editor |
The first two feel like engineering because they involve typing. Only the third one changes what's possible.
Bug Two: The Cap That Cut Two Thirds of a Person Out of One Prompt
The apply engine sends a resume to a model more than once per application. One prompt writes the tailored resume. Another answers the employer's screener questions, and that one is grounded: it can only answer from what's in the document, and its own instructions say that if nothing of the kind is listed, give an honest No.
The tailoring prompt sliced the document at 60,000 characters. The screener prompt sliced the same document, in the same run, at 14,000.
Nobody decided that. Two people picked a number that felt safe for the model they had in front of them, and both of those people were me, months apart.
The test fixture fit inside the smaller cut
The fixture resume was 10,700 characters. Comfortably under both caps. Every test passed, for months, honestly.
The real document was 53,045 characters. At 14,000, two thirds of the person went invisible. Skills, Education, Certifications, and four of the five employers, all below the cut line.
The prompt then did exactly what it was told. It couldn't see a degree, so it followed its own honest-No rule and denied a degree the applicant holds. Calmly. In an application that got submitted.
That's the worst failure mode I've hit. Not a crash. Not even a model making something up. A correct answer to a question I had quietly changed without telling anyone, including myself.
Then it happened again, one number later
I fixed the 14,000. Set every prompt to the same 60,000. Felt pretty good about the afternoon.
The production resume is 63,061 characters. Three prompts capped it at 60,000. So 3,061 characters had been vanishing from every single application since the day that file was uploaded, and the fix I had just shipped was the thing doing it.
Same defect, one number later. When a bug comes back with a different constant, the constant was never the bug.
The fix: one function, and it says when it bites
Every cap now goes through one function, capGrounding. Nothing calls .slice() on a person's document anywhere else in the codebase. One limit, one place, every consumer.
The part that matters more than the limit is the logging. It logs the moment it actually cuts. Not that a limit exists. Not that a limit was consulted. That this document, in this run, lost this many characters, and it collects every cut into a report at the end of the run.
A limit that trims nothing should be silent. A limit that removes most of the input should be the loudest line in the log.
The Test Fixture Is Working Against You
Fixtures get chosen for being convenient. Small, clean, quick to read in a diff, fast in CI. Every single property that makes a fixture pleasant to work with is a property that hides a size bug.
Mine was 10,700 characters because a 53,000 character file is miserable to scroll past in a pull request. That preference, and nothing else, is why a real person's education disappeared.
What I do now: every suite gets one fixture that's deliberately bigger than every cap in the system, and one that sits just over a boundary. The 63,061 case is a fixture now. It exists to fail on the day somebody sets a limit back to a round number.
Round numbers are a smell. 14,000. 60,000. 1,000. Nobody measured any of those. Somebody typed them.
Bug Three: The Read That Stops at 1,000 Rows and Reports Success
The apply engine's data sits in Postgres behind PostgREST. A plain select returns at most 1,000 rows by default. It doesn't error. It doesn't warn. It hands you 1,000 rows and a 200, and the client library is delighted.
While your table has fewer than a thousand rows this is invisible. It stays invisible right up until the day it isn't.
The first time, it cost a duplicate application
Deduplication worked by reading everything already applied to and checking each new job against that set. Once the applied table crossed a thousand rows, the check started reading a slice of my history and comparing against the slice. So the engine applied to the same posting twice.
The dedupe logic was correct the whole time. Its input quietly stopped being the truth.
The second time, it made me debug a run that was fine
Wave 52 picked zero jobs. That looks like a real defect, so I wrote a quick control query to see how many employers were still free to apply to. It came back with 209. That number said the wave should have found plenty, so I went off hunting for a bug in the picker.
There was no bug in the picker. My control query hit the same ceiling. Paged properly, the real figures were 1,213 applied and 1,170 submitted, and the wave had been right all along. The check was the broken thing.
That's the second cost of a silent truncation and it's the more expensive one. The first time it corrupts your data. The second time it corrupts your judgment, and you spend an afternoon rewriting code that was already correct.
The fix
Every read against that database is paged, no exceptions, and the paging loop lives in one helper so there's nothing to remember at the call site. Where a count is all I need, I ask for a count instead of asking for rows and measuring the array.
A number that comes back exactly equal to a limit is not a measurement. It's the limit.
Three Bugs, One Shape
| The bug | What it did | What it returned | What found it |
|---|---|---|---|
| Whole-row write plus explicit null | Erased computed values on thousands of lots, hourly | Success, every hour | A doc claiming 65 percent next to a page showing almost none |
| Two caps on one document | Removed two thirds of a resume from one prompt | A calm, wrong answer | Reading a submitted application by hand |
| Unpaged read | Returned 1,000 rows out of thousands | 200 OK | A duplicate, then a wasted afternoon |
Look at the last column. Not one of these was found by the software. They were found by a human noticing that two things which should agree didn't.
And every one of them had the information at the exact moment of the failure, then threw it away. The importer knew it was writing null over a value that was already sitting there. The cap knew it had just removed most of the document. The read knew it had hit the ceiling. All three had it, none of them said it.
What I Actually Changed
Make the bad state unrepresentable
If a rule can only be kept by remembering it, it will be broken by the next person, and the next person is you in March. Push the rule into a type, a union, an exhaustive switch, a constructor that won't build the wrong object. A build error is the cheapest alert you will ever install.
Log the moment a limit bites
Every truncation, every cap, every batch boundary, every retry ceiling. Not "limit is 60,000" at startup, which tells you nothing. "Cut 3,061 characters from this document in this run." A limit that never fires costs one silent branch. A limit that fires and stays quiet costs you a month.
Never trust a read you didn't page
Assume every client and every API truncates. Put the paging in one helper and use it everywhere, including in the throwaway query you wrote to check whether something else is broken. Especially that one.
Name the test after the failure
Not test_upsert_lot. Something a person can read six months later and understand what went wrong the first time. My favourite test in either codebase is called "a re-import that omits parcel and value does not erase them." It reads like a small tombstone, which is roughly what it is.
A Short List You Can Run Against Your Own Code Today
- Grep for
.slice(andsubstring(on anything that came out of a user's document or a model's context. Every hit needs a single shared limit and a log line. - Grep for round numbers in constants. 1000, 10000, 60000, 100000. Ask who measured each one.
- Find every write that spreads a whole object back into an upsert. Then find who else writes those columns.
- Find every read with no explicit range or page loop. Compare its result length to the client's default ceiling.
- Find every success log that can't tell you how much it processed. "Done" is not a log line. "Done, 1,514 rows, 0 truncated" is.
Half an hour, five greps. I'd bet you find at least one.
Ready to Write Yours Down?
The most useful thing I did after all three was write the post-mortem while it still stung, then put the rule that came out of it somewhere the compiler could enforce it.
If you draft yours in markdown and need to hand it to somebody as a clean PDF, we have a free converter for that.
Try the Markdown to PDF Converter Now →
One Last Thing
Your error tracker is an honesty machine for the code that admits it failed.
The code that quietly does the wrong thing and returns 200 has no such machine, and it never will, because nothing is wrong from where it's standing.
So the work sits upstream of monitoring. Delete the states you don't want. Make the system say something the instant it takes a shortcut, and never believe a read you didn't page.
Three defects, no stack traces, weeks of damage between them. Every one was preventable with a type, a log line, or a loop.
The bug that pages you at 3am is not your problem. The one that lets you sleep is.