Building an On-Call Culture from Scratch at a "Move Fast, Break Things" Startup

We were a 7-person engineering team at a seed-stage B2B SaaS startup. There was no on-call rotation — when things broke, the CTO would get a text from a customer and scramble to fix it. Deploys happened whenever anyone pushed to main. There were no runbooks, no incident channels, and no post-mortems. The engineering culture was very much "ship fast and figure it out." This worked when we had 5 customers but we had just signed our 50th and the cracks were showing.

The tipping point was when we lost a $200k ARR enterprise customer because a database connection leak caused 4 hours of downtime on a Friday night and nobody noticed until Monday. The CTO called an all-hands and we decided to build an on-call culture. The challenge was doing this without killing the startup speed that got us to 50 customers. We introduced PagerDuty with a simple 1-week rotation among all 7 engineers. We wrote runbooks for the top 10 most common issues. We created a #incidents Slack channel with a lightweight post-mortem template.

Within 6 months, our MTTR went from ~4 hours to 22 minutes. We went from 3-4 customer-reported incidents per month to near-zero — we were catching and fixing issues before customers noticed. The biggest surprise was that shipping speed actually increased because engineers had more confidence in deploys when they knew there was a safety net. We kept the culture lightweight — the post-mortem template is 5 bullet points, not a 10-page document.