{"report":"incidents","url":"https://prizepicksapi.com/incidents","schema_version":"1.0","scope":"Incidents with a written record since collection began on 27 October 2026. Times are UTC.","count":8,"open_or_monitoring":2,"incidents":[{"id":"2026-11-03-board-coverage-timeouts","date":"2026-11-03","title":"Board coverage reads timed out under database load","area":"status","status":"monitoring","started_at":null,"ended_at":null,"impact":"The status page showed board coverage as unavailable while the WebSocket stayed healthy. Authenticated coverage and compact projection reads returned 503 in a post-release smoke test, with no credit charged. Lines kept being collected.","cause":"The status page ran the coverage aggregate inside the visitor request. The same aggregate was cancelled after about 9.7 seconds in a post-release check, with the database host under load.","fix":"Coverage is now measured on a schedule by one bounded background reader. The status page reads the stored row, shows its measurement time and marks it stale when the refresh falls behind.","recovery_note":"Authenticated read latency remains under observation.","recovery_seconds":null},{"id":"2026-11-02-comparison-server-errors","date":"2026-11-02","title":"Comparison and projection requests returned server errors","area":"api","status":"monitoring","started_at":"2026-11-02T23:53:00Z","ended_at":"2026-11-03T00:02:00Z","impact":"22 comparison and 4 projection server errors were recorded in the 48 hours before 05:29 UTC on 3 November. Some failed calls took more than two minutes. The latest projection failure was at 00:59 UTC.","cause":"Database statements were cancelled during the same period.","fix":"Fast connection failures on comparison reads retry once. Identical concurrent reads share one query. Server errors carry Retry-After and a correlation ID. The health check fails on three API server errors in ten minutes.","recovery_note":"The recorded comparison window lasted 9 minutes. Later samples still show variable latency, so this stays under observation.","recovery_seconds":540},{"id":"2026-11-02-summary-snapshot-stale","date":"2026-11-02","title":"Dashboard summary kept an old snapshot for three hours","area":"status","status":"resolved","started_at":"2026-11-02T23:03:00Z","ended_at":"2026-11-03T02:03:00Z","impact":"The dashboard summary showed figures from 23:03 UTC until 02:03 UTC, with their real timestamp. It showed no invented freshness and no zeros.","cause":"The summary depended on the full homepage refresh, which hit database timeouts.","fix":"An independent, bounded summary refresh now runs every two minutes.","recovery_note":"Full refreshes succeeded again from 02:03 UTC, three hours after the last good snapshot.","recovery_seconds":10800},{"id":"2026-11-02-stream-import-backlog","date":"2026-11-02","title":"First stream deployment could not keep up with its imports","area":"collection","status":"resolved","started_at":null,"ended_at":null,"impact":"On its first day the new WebSocket feed queued batches: a 520 ms import could not service 250 ms batches. Pending batches stayed on disk and were transferred with their checkpoint. REST collection runs separately from the stream.","cause":"The stream importer ran in a different region from the database, so every transaction crossed regions.","fix":"The stream importer moved next to the database. The first healthy sample after the move showed a 12 ms import, 261 ms from receipt to database and an empty queue.","recovery_note":"Healthy production samples were recorded by 04:23 UTC on 2 November.","recovery_seconds":null},{"id":"2026-11-01-cached-board-restores","date":"2026-11-01","title":"Cached all-leagues copies restored lines already removed","area":"data","status":"resolved","started_at":null,"ended_at":null,"impact":"A capped all-leagues board up to about 15 minutes old could bring back a line a newer complete board had removed, and could overstate freshness. A replay of 192 real boards measured 1,004 such restores and 832 seconds of overstated freshness. History written before the fix was left unchanged.","cause":"The all-leagues response is a cache whose copies can go backward. Ingest treated it as current.","fix":"A copy older than the latest complete board of a league can no longer restore a line or clear a first miss. Its rows carry the source time, not the fetch time. The same replay afterward measured 0 and 0.","recovery_note":"Found on 1 November, US time. The fix merged at 02:15 UTC on 2 November.","recovery_seconds":null},{"id":"2026-10-28-database-transfer-quota","date":"2026-10-28","title":"Hosted database locked on its transfer quota; production moved to its own database","area":"collection","status":"resolved","started_at":"2026-10-28T12:34:25Z","ended_at":"2026-10-28T20:32:00Z","impact":"The live API, stream and collection were down. Nothing was collected from 12:34 to 18:54 UTC, and those lines will never exist in the history. From 18:54 UTC boards were saved to disk and replayed afterward with their original times, flagged as backfilled and never sent to webhooks. The sitemap returned 500 while the database was locked.","cause":"Reading full payloads on every board used about 18 GB of network transfer against a 5 GB monthly allowance on the hosted plan.","fix":"Production moved to its own database with nightly off-machine backups and a verified restore. Ingest reads hashes instead of full payloads. Fetching now continues to a disk spool during a database outage, and out-of-order boards take a history-only path.","recovery_note":"About 8 hours from the last stored collection to the first live collection on the new database. Replay from disk continued after that.","recovery_seconds":28655},{"id":"2026-10-28-checkout-empty-catalog","date":"2026-10-28","title":"Checkout returned 503 after the database move","area":"billing","status":"resolved","started_at":null,"ended_at":null,"impact":"Starting a checkout returned 503 because the billing catalog on the new database was empty.","cause":"The catalog table was not carried over in the database move.","fix":"The catalog now rebuilds itself from the payment processor by exact product, amount and interval. A live checkout session was verified afterward.","recovery_note":"Fixed on 28 October.","recovery_seconds":null},{"id":"2026-10-28-cached-copy-history","date":"2026-10-28","title":"Stale cached boards wrote false line moves into history","area":"data","status":"resolved","started_at":null,"ended_at":null,"impact":"Cached board copies with an older source time looked like a line moving back and forth. 1,251 false history rows and events across 486 offers were written and then removed.","cause":"Ingest compared boards by arrival, not by source update time.","fix":"Copies whose source time goes backward are skipped at ingest, and a repair script removed the rows. No stale regression was seen after the repair.","recovery_note":"Found and repaired on 28 October.","recovery_seconds":null}]}