My Vibecoded Server Went Down
An AI wrote the app in six weeks and the app was fine. Everything that broke was around the code.
On August 3, every site I run went down at once, came back twelve minutes later, and did it again. About twenty-one times, over sixteen hours. What knocked them over was the least important thing I own: a public directory I built in six weeks of prompting, 203 commits, almost none of them typed by hand.
The generated code was never the problem. Twenty-seven requests a second took down five production sites, and the bug was a certificate key type I never chose, sitting behind a CPU limit I wrote into my own notes months earlier and never reopened. Everything that failed sat around the code, in the defaults I never opened.
#The five rules
- Isolate first, and keep the parts small. Anything public and unpredictable gets its own machine before it gets traffic. A second box was a rounding error on the monthly bill. Sixteen hours of five domains down was not.
- Decide the defaults you inherited. The one most dangerous to skip. A generated app arrives with a certificate key type, a CPU ceiling, a cache with no eviction policy and a placement, none of which anybody chose. They are still your production decisions.
- Make the thing cheap before you try to make the traffic small. An index that carries its own sort order and a cached render bought more headroom than a day of throttles, and they keep working when the throttles get bypassed.
- Buy observability before you need it. Alarm on saturation rather than on errors. My outage produced no errors at all: the proxy stopped accepting connections, so there was nothing to log.
- Treat every interface you publish as a contract. robots.txt, the filter URL space, the apex redirect, the certificate key type. Every failure in this story is one I shipped without reading it back.
A default you never chose is still a decision you shipped.
I did not block a single crawler. 99.6% of the requests to my public pages were machines, and a directory that blocks crawlers is a directory nobody finds. What shipped was a contract instead of a wall: six filter parameters disallowed in robots.txt, ten AI crawlers named and allowed by name, and one cap on how many renders may run at once.
Related: catalog.agentage.io - the directory that fell over, now on its own box · agentage.io - the memory product that was sharing the machine with it · @vreshch on GitHub.