How a read-only analytics API went from tens of seconds to low single-digit seconds at p99, and how the storage behind it shrank by roughly three quarters.
A read-heavy analytics API was spending its time moving data, not computing. The fix was architectural, not a tune-up: embed DuckDB inside the Rust service and query local columnar files instead of a remote database.
What changed
- Parquet snapshots on object storage, pulled to local disk on a schedule — no network round trips on the query path.
- DuckDB's vectorized engine does aggregations and joins far faster than row-by-row fetching.
- The storage footprint dropped by roughly three quarters thanks to columnar compression.
What it cost
- Reads are only as fresh as the last snapshot — fine for analytics, wrong for transactional data.
- Memory must be budgeted explicitly; an embedded engine shares the process.
The lesson: when reads dominate and data changes in batches, moving the engine to the data beats tuning the network path. Measure p99 before and after — the averages will undersell the win.