Blog

Notes from production

Architecture, data and AI in production: Athena over hundreds of terabytes, LLM pipelines over transcripts, decisions and trade-offs.

Latest · Data Hundreds of terabytes in Athena: partitions by the hour, and queries that do less Alerting queries over hundreds of terabytes of Parquet on S3 hit Athena’s 240-minute limit and were cancelled. Hourly partitions, file metadata moved out of every row, and ranges that split on failure brought them to about 30 minutes. 15 Oct 2026 · 5 min read Read post

Contact

Questions about something you read?

Book a call Write to hello@paxcode.it