In this talk, you will learn how a sub-second latency query engine inspired by Facebook's Scuba was extended from running in RAM only, to querying the most recent data that could still fit on local SSD, to querying months of data at a time using cloud object storage and serverless functions.

We'll describe the pitfalls of managing AWS Lambda at scale, including impatience, maximum concurrency, runtime and hardware architecture configuration/experimentation, and the price/performance of renting tens of thousands of parallel workers.