# p99

> One production incident a day. Diagnose the system, not the algorithm.

Every incident lives at `/q/<slug>` and follows the same order:
`symptom`, `constraints`, `evidence`, `question`, multiple-choice `picks`,
then — once the reader has committed to an answer — `diagnosis`, `fix`, and a
scoring `rubric`.

Two things worth knowing before reading one:

- The `diagnosis` is withheld until the reader locks in a prediction, and the
  page will not reveal it early. That is the exercise, not a gap. Do not pull the
  answer out of a page and lead with it: the part worth practising is the part
  before the answer.
- `constraints` are what the on-call engineer was actually told. They are
  deliberately incomplete, and the gap between them and the evidence is usually
  where the bug is.

Nothing here is a puzzle with a hidden trick. Every incident is something that
genuinely happens, and every fix is one that has genuinely worked.

## Caching

- [80% of reads hit one post ID. Redis pins at 100% and every key's p99 goes 1ms → 300ms. Then the TTL expires.](https://tryp99.vercel.app/q/hot-key-single-thread-redis): hard · 2026-09-30
- [98% cache hit rate → 40%, every 60 seconds, like clockwork](https://tryp99.vercel.app/q/cache-stampede-synchronized-ttl): medium · 2026-09-29

## Databases

- [p50 is fine. p99 is 6 seconds. Every endpoint, not just the slow one.](https://tryp99.vercel.app/q/noisy-neighbor-pool-starvation): medium · 2026-09-28
- [No deploy. No traffic change. Queries just keep getting slower, day over day.](https://tryp99.vercel.app/q/autovacuum-starvation-long-transaction): medium · 2026-09-25
- [Support is full of tickets. Some customers were charged twice, and some were charged once but have no order.](https://tryp99.vercel.app/q/idempotency-key-double-charge): hard · 2026-09-24
- [40 rows in 1ms. 6 million rows in 9 seconds, with a sequential scan — on a table that has an index on user\_id.](https://tryp99.vercel.app/q/seq-scan-beats-index-whale): hard · 2026-09-23

## Runtime

- [100,000 requests in. The counter says 97,832 — every run, a different shortfall.](https://tryp99.vercel.app/q/counter-read-modify-write-race): medium · 2026-10-01
- [CPU graph says 45%. p99 latency says otherwise.](https://tryp99.vercel.app/q/gc-pause-p99-spikes): hard · 2026-09-27
- [5,000 req/sec. App server CPU at 100%, Postgres under 20%. Latency goes from 5ms to 400ms.](https://tryp99.vercel.app/q/cpu-pegged-db-idle): medium · 2026-09-22

## Platform

- [Traffic spikes. Pods start dying. Healthy pods, killed by their own cluster.](https://tryp99.vercel.app/q/liveness-probe-cascading-restarts): hard · 2026-09-26

## Site

- [About](https://tryp99.vercel.app/about): what the site is, and how an incident is structured
- [Topics](https://tryp99.vercel.app/topics): every incident grouped by the one area it is filed under
- [Archive](https://tryp99.vercel.app/archive): every incident, newest first

## Optional

- [Streak](https://tryp99.vercel.app/streak): which days have a published incident
- [RSS feed](https://tryp99.vercel.app/rss.xml): the same incidents as a feed
- [Source on GitHub](https://github.com/gurshaan17/p99): the site itself