Board coverage reads timed out under database load
Monitoring
Status
Impact
The status page showed board coverage as unavailable while the WebSocket stayed healthy. Authenticated coverage and compact projection reads returned 503 in a post-release smoke test, with no credit charged. Lines kept being collected.
Cause
The status page ran the coverage aggregate inside the visitor request. The same aggregate was cancelled after about 9.7 seconds in a post-release check, with the database host under load.
Fix
Coverage is now measured on a schedule by one bounded background reader. The status page reads the stored row, shows its measurement time and marks it stale when the refresh falls behind.
Recovery
Authenticated read latency remains under observation.
Comparison and projection requests returned server errors
Monitoring
APIFrom 2026-11-02 23:53 UTCTo 2026-11-03 00:02 UTCRecovery time 9 min
Impact
22 comparison and 4 projection server errors were recorded in the 48 hours before 05:29 UTC on 3 November. Some failed calls took more than two minutes. The latest projection failure was at 00:59 UTC.
Cause
Database statements were cancelled during the same period.
Fix
Fast connection failures on comparison reads retry once. Identical concurrent reads share one query. Server errors carry Retry-After and a correlation ID. The health check fails on three API server errors in ten minutes.
Recovery
The recorded comparison window lasted 9 minutes. Later samples still show variable latency, so this stays under observation.
Dashboard summary kept an old snapshot for three hours
Resolved
StatusFrom 2026-11-02 23:03 UTCTo 2026-11-03 02:03 UTCRecovery time 3.0 h
Impact
The dashboard summary showed figures from 23:03 UTC until 02:03 UTC, with their real timestamp. It showed no invented freshness and no zeros.
Cause
The summary depended on the full homepage refresh, which hit database timeouts.
Fix
An independent, bounded summary refresh now runs every two minutes.
Recovery
Full refreshes succeeded again from 02:03 UTC, three hours after the last good snapshot.
First stream deployment could not keep up with its imports
Resolved
Collection
Impact
On its first day the new WebSocket feed queued batches: a 520 ms import could not service 250 ms batches. Pending batches stayed on disk and were transferred with their checkpoint. REST collection runs separately from the stream.
Cause
The stream importer ran in a different region from the database, so every transaction crossed regions.
Fix
The stream importer moved next to the database. The first healthy sample after the move showed a 12 ms import, 261 ms from receipt to database and an empty queue.
Recovery
Healthy production samples were recorded by 04:23 UTC on 2 November.
A capped all-leagues board up to about 15 minutes old could bring back a line a newer complete board had removed, and could overstate freshness. A replay of 192 real boards measured 1,004 such restores and 832 seconds of overstated freshness. History written before the fix was left unchanged.
Cause
The all-leagues response is a cache whose copies can go backward. Ingest treated it as current.
Fix
A copy older than the latest complete board of a league can no longer restore a line or clear a first miss. Its rows carry the source time, not the fetch time. The same replay afterward measured 0 and 0.
Recovery
Found on 1 November, US time. The fix merged at 02:15 UTC on 2 November.
Hosted database locked on its transfer quota; production moved to its own database
Resolved
CollectionFrom 2026-10-28 12:34 UTCTo 2026-10-28 20:32 UTCRecovery time 8.0 h
Impact
The live API, stream and collection were down. Nothing was collected from 12:34 to 18:54 UTC, and those lines will never exist in the history. From 18:54 UTC boards were saved to disk and replayed afterward with their original times, flagged as backfilled and never sent to webhooks. The sitemap returned 500 while the database was locked.
Cause
Reading full payloads on every board used about 18 GB of network transfer against a 5 GB monthly allowance on the hosted plan.
Fix
Production moved to its own database with nightly off-machine backups and a verified restore. Ingest reads hashes instead of full payloads. Fetching now continues to a disk spool during a database outage, and out-of-order boards take a history-only path.
Recovery
About 8 hours from the last stored collection to the first live collection on the new database. Replay from disk continued after that.
Checkout returned 503 after the database move
Resolved
Billing
Impact
Starting a checkout returned 503 because the billing catalog on the new database was empty.
Cause
The catalog table was not carried over in the database move.
Fix
The catalog now rebuilds itself from the payment processor by exact product, amount and interval. A live checkout session was verified afterward.
Recovery
Fixed on 28 October.
Stale cached boards wrote false line moves into history
Resolved
Data
Impact
Cached board copies with an older source time looked like a line moving back and forth. 1,251 false history rows and events across 486 offers were written and then removed.
Cause
Ingest compared boards by arrival, not by source update time.
Fix
Copies whose source time goes backward are skipped at ingest, and a repair script removed the rows. No stale regression was seen after the repair.