My auto-apply engine had filed 1,183 job applications end to end, and every single one of them was mine. I read that as proof the product worked. Then I ran one query about everyone else who had pressed the same button, and found out I had spent months testing on the only person in the database who could possibly succeed.
The Number I Was Proud Of
Between the middle of August and the first of September I ran 52 waves of applications in 18 days. The engine opens the employer's real application form, fills the identity and legal fields from what I told it once, answers the written questions using only my own resume, attaches a resume written for that posting, and submits. As of September 1st it had filed 1,183 applications for me, end to end.
That is a real number and I still like it. It's just not the number I should have been staring at.
Because those 1,183 runs all came from one account. Mine. The account with every field filled in, every connection wired up, every legal question answered, because I filled them in one at a time over months while I was building the thing that reads them.
Then I Looked at Everybody Else
Here is the query I should have written on week one.
Not "how many applications did the system file." That question flatters me. The right question is "how many people other than me have ever completed a run."
The answer, on August 31st: nine accounts had ever tapped Apply. One of them was me. Of the eight strangers who tried, the result was 30 attempts and 30 failures. Blocked or failed, every one, no exceptions.
Thirty out of thirty. Against a system I had personally watched succeed more than a thousand times.
If your aggregate success number and your per-user success number disagree, the aggregate is lying to you and the per-user one is telling you what your product actually is.
Thirty Failures, Zero Surprises
The part that made me put my coffee down was not the failure rate. It was what happened when I grouped the failures by cause.
Twenty of the thirty died on the same missing piece of account setup, a step that exactly one account in the entire database had ever completed. Mine. The other ten died on required fields that were simply empty when the form asked for them.
Every one of those thirty failures was knowable before a browser ever opened. Not knowable with a clever heuristic or a machine learning model. Knowable from rows the server already had in front of it, sitting in Postgres, at the moment the person clicked the button.
The system was capable of knowing that a run could not possibly succeed, and it launched the run anyway. Then it charged for it. Then it waited about three minutes. Then it came back with a refusal the person had no way to interpret.
Why the Founder Is the Worst Possible Test
I want to make this general, because I don't think this is my bug. I think it's a founder bug.
You built the product. That means you have the most complete profile in your own system, by a wide margin, and you got it that way accidentally. Every time you added a field you filled it in to test the field. Every time you added an integration you connected it to see if the integration worked. Every setup step exists in your account because you wrote the setup step.
So the founder is the one user who never experiences the empty state, never hits the missing prerequisite, and never sees the confusing error. You are running the product in a configuration no real person will ever reach.
Worse, you interpret your own success as evidence. I did. Eleven hundred applications felt like proof. It was proof of exactly one thing: that the engine works when it is handed a complete profile. It said nothing at all about whether anyone else could hand it one.
The Actual Shape of the Gap
Once I stopped counting applications and started counting people, the picture took about ten minutes to assemble. Out of 1,532 registered accounts at the end of August:
| What the run needs | Accounts that had it |
|---|---|
| An apply profile of any kind | 58 |
| A location on that profile | 29 |
| A LinkedIn URL | 17 |
| Legal work declarations answered | 10 |
| Education filled in | 1 |
Read the bottom row again. One person in fifteen hundred had filled in education, on a product whose whole job is to answer employer questions on your behalf.
There was no mystery here. There was no deep funnel analysis. It was five counts against five columns, and I could have run it in month one.
The Check Already Existed. Nothing Enforced It.
This is the detail that still annoys me.
The system already computed a thing called profile gaps, with a severity level, and one of the severities was literally "blocks." It had been there from the beginning. It rendered a yellow banner in the web UI.
That was the entire enforcement. A yellow banner. Zero server-side checks. The API would happily accept an application request from an account that its own code had already classified as unable to complete one.
I had built the knowledge and then thrown it away at the door. That's the whole bug in one sentence.
What Failing Slowly Actually Costs
It's tempting to file this under "poor error messages." It isn't. Failing slowly costs a new user three things at once, and they compound.
It costs them time. Three minutes of watching a progress indicator, which for a first-time user is the entire trial period. Nobody gives you a second three minutes.
It costs them money. In my case, credits, spent on a run that had zero chance from the first millisecond.
It costs them the ability to fix it. A refusal that arrives after a long wait and doesn't name the missing field teaches the person nothing. They can't tell whether the product is broken, whether they're broken, or whether the employer rejected them. So they leave, and my day-one retention numbers said exactly that: 2 percent. Of 442 accounts over 90 days, nine ever signed in on a later day.
The Fix Was a Refusal
I shipped it that night, August 29th. A readiness gate.
Before anything else happens, the server checks the account's profile against what this particular run will need. If something required is missing, the request is refused immediately, no browser launched, no credits spent, and the response names the missing piece in plain language and tells the person where to go fix it.
The best feature I've built into that engine is a feature that declines to do the thing. It's a door that says no.
I keep coming back to how backwards that felt while I was writing it. Every instinct says ship capability, ship more automation, make it do more. The thing that actually made the product usable by a second human being was making it do less, faster, and say why.
Two Doors, Not One
The first version of the gate had one check, sitting in the HTTP handler that accepts an apply request. An adversarial review of my own patch killed it.
The reason: the queue can get rows through a second path. When the container restarts, boot recovery re-enqueues whatever was in flight, and that path never touches the HTTP handler. So an application I had correctly refused at the front door would have quietly run anyway at the next deploy.
The gate ships with two doors. One in the request handler, one in the queue right before the browser starts. Both of them refuse. The second one exists because I assumed my own first fix was complete and it wasn't.
Any check that lives on only one code path is a check you'll discover missing during an incident.
Two Hours Later It Caught a Real Person
The gate went out that night. About two hours after deploy, the eighth non-owner account tapped Apply.
The row that came back had credits used: 0, and a message naming exactly the setup step that was missing and what to do about it. The person got a straight answer in a couple of seconds instead of a three-minute wait ending in something incomprehensible.
That was the first time the fix had been proven on a human being rather than on my own test request. Two hours. After months of the thing quietly working for nobody but me.
How to Find This in Your Own Product
If you're one person building something, here's the version of this you can run this week. It's three moves and none of them take long.
1. Count completions by user, never in aggregate
Your total number is the sum of your own usage plus everyone else's, and if you're a heavy user of your own product, your usage swamps the signal completely. Write the query that groups by user id and counts successful completions. Then delete your own row from the result and look at what's left.
If what's left is a column of zeroes, you have my bug, and no amount of aggregate growth is going to tell you.
2. Walk the funnel as somebody who isn't you
Pull the counts for every prerequisite your core action depends on. Not "how many users do we have," but "how many users have the specific things this action requires, one column at a time."
I did this as five counts in a table and the answer was obvious on sight. Sign up with a genuinely fresh account too, on a device where you aren't logged in, and try to complete your core action without touching the database. Whatever you reach for to unstick yourself is the thing your users can't reach for.
3. Instrument every refusal, the same way you instrument a success
This is the part people skip. When you add a check that blocks an action, log why it blocked. Store the reason on the row. Then you have a table that tells you, in ranked order, what your users are missing, and that table becomes your onboarding roadmap without you having to guess at it.
Refusals are the highest-value events in a product and almost nobody records them. Successes tell you the happy path exists. Refusals tell you where everyone else is standing.
While You're In There, Check Your Analytics Too
A warning from the same week, because it's the same disease.
I went looking for signup numbers to sanity check all of this and found my analytics tool had recorded exactly 1 signup for the month of August. The database had 147. The event was firing on a path most signups never took.
Same week, same category: my payments provider never wrote subscription status back to the database, so every one of my 1,532 user rows said "not a paying customer" while ten people were actually paying me every month.
If you haven't reconciled your analytics against your database in a while, do that before you trust any dashboard in this post's advice. I was making decisions on numbers that were wrong in both directions.
What I'd Do Differently
I'd write the refusal before I wrote the capability. On day one, before the feature works at all, ship the check that says "you can't do this yet, here's what's missing." Then build the thing behind it.
That ordering feels absurd when there are zero users. It stops feeling absurd the first time somebody who isn't you presses the button.
The month I lost isn't really the month between shipping the engine and shipping the gate. It's every day in that month when I looked at a growing application count and believed it meant something about other people.
Ready to Stop Guessing What's Missing?
Same idea, smaller scale: before you send a resume anywhere, run it through a check that names what's missing instead of letting you find out three weeks later through silence.
Try the ATS Resume Scanner Now →
It's free, it takes about a minute, and it'll tell you which required things your resume doesn't have. Which is exactly what I should have built for my own users a month earlier than I did. If you want the applying part automated after that, Jobbi is where the engine from this post actually lives, on iOS and web.
One Last Thing
Your product working for you is not a signal. It's the default state of every product that has ever been built by the person using it.
The only number that means anything is how many people who aren't you got all the way to the end.
Go count that one. It takes ten minutes and it'll ruin your week in the most useful way possible.
Then build the door that says no, and make it say why.
Written by Evgeniy Poznyak, who builds things at Softery.io and mostly writes about the parts that went wrong.