How I Cut S3 Storage Costs by 60% by Accident — PostgreSQL Backups, CSV, and the Parquet Revelation

This one wasn't planned. No ticket, no objective, no cost optimization sprint. I was pulling PostgreSQL backup data into a local analytics project — just exploring, building some charts — and I happened to run the numbers on file size across formats. The result made me stop and check whether I'd made an error.

I hadn't. Parquet was just that much smaller.

The Setup

The project was simple: take some production data from a PostgreSQL backup, run analytics on it, build visualizations. The data lived in PostgreSQL. The backups were already being exported to S3 as CSV — standard practice, nothing fancy. I pulled the CSVs locally to work with them.

At some point during the exploration, I converted one of the CSV exports to Parquet using pandas just to see if querying would be faster. It was. But what caught my attention was the file size before I even ran a query.

The CSV was sitting at several hundred megabytes. The Parquet file — same data, same rows — was roughly a third of that. I ran it again on a larger table. Same story, bigger delta.

The Numbers

I started benchmarking more carefully across the tables I had. The compression ratio landed consistently between 3x and 5x — Parquet files were 3 to 5 times smaller than their CSV equivalents for the same PostgreSQL data.

This isn't surprising in retrospect, but I hadn't internalized it before seeing it on my own data. Here's why the gap is that large:

• Columnar storage. Parquet stores data column-by-column rather than row-by-row. For analytical data, columns tend to be repetitive — status fields, category codes, date ranges. Columnar layout exposes this repetition directly to the compression algorithm.
• Dictionary encoding. Before compression even runs, Parquet applies dictionary encoding on low-cardinality columns. A column with values like "active", "inactive", "pending" gets encoded as integers — 0, 1, 2 — before Snappy or ZSTD touches it.
• No structural overhead per row. CSV pays a cost for every delimiter, every newline, every repeated column name implicitly encoded in position. Parquet pays schema overhead once at the file level.

The VLDB 2024 paper "An Empirical Evaluation of Columnar Storage Formats"^[1] formalizes this: columnar formats achieve 5–10× compression ratios typically, with low-cardinality columns reaching 20:1 or higher. My PostgreSQL data — with its status fields, merchant categories, and date columns — sat right in the middle of that range.

For reference: a DuckDB TPC-H benchmark at scale factor 20 showed Parquet at 3.2 GB vs CSV at 16 GB^[2] — a 5× gap. Even gzip-compressed CSV remained 2–3× larger than Parquet with Snappy compression.

From Curiosity to Production

Once I saw the numbers, the question was obvious: why are we exporting PostgreSQL backups to S3 as CSV?

The answer was inertia. CSV is easy to inspect, works everywhere, requires no library to open. Those are real advantages for debugging. But for data that's being stored at scale and queried analytically, they don't outweigh the cost of keeping files 3–5× larger than they need to be.

Crunchy Data's work on incremental PostgreSQL-to-Parquet archival^[3] confirmed the pattern: structured PostgreSQL data exported to columnar formats consistently achieves significant storage reduction without any loss of fidelity. Their recommendation was the same conclusion I'd arrived at — for analytical archival, Parquet is the right default.

The migration was straightforward. The export pipeline was already in Python. Swapping df.to_csv() for df.to_parquet(compression='snappy') was a one-line change. The harder part was validating that the data round-tripped correctly — row counts, nulls, type fidelity — which it did.

The S3 Cost Impact

S3 storage pricing is simple: you pay per GB stored. If your files are 3–5× smaller, your bill is 3–5× smaller for that data, plus reduced data transfer costs on every read.

In practice, the compression ratio on our specific data averaged around 3.2×, which translated to approximately 60% reduction in S3 storage costs for the backup data we migrated. The number is consistent with what CloudForecast^[4] and Sedai^[5] report as typical for structured data migrations to Parquet: 60–80% storage reduction is the normal range.

There's a secondary benefit that compounds over time: Snappy-compressed Parquet is also faster to query analytically than CSV — 7–10× faster in Crunchy Data's benchmarks^[3] — because columnar reads allow skipping irrelevant columns entirely. For a pipeline doing analytics on top of S3, this matters.

Codec Choice

One decision worth noting: compression codec. Parquet supports Snappy, GZIP, ZSTD, and LZ4. The right choice depends on your read/write pattern:

• Snappy — fast compression and decompression, moderate size reduction. Good default for frequently queried data.
• ZSTD — better compression than Snappy (15–20% smaller files) with minimal performance cost^[6]. Best for archival data that's read less frequently.
• GZIP — best compression ratio, slowest. Only worth it for cold storage that's rarely touched.

For backup data on S3 that's queried periodically, ZSTD is the right call. For data queried daily in hot analytical pipelines, Snappy wins on latency.

The Takeaway

I didn't set out to optimize S3 costs. I was playing with data for an analytics project and accidentally benchmarked two file formats side by side. The gap was large enough that it immediately changed how I thought about every CSV sitting in object storage.

The math is simple: if you have structured tabular data in S3 as CSV and you're not using Parquet, you're paying for 3–5× more storage than you need to. The migration is a few lines of Python. The savings compound with every GB you store and every byte you transfer.

Sometimes the best optimizations aren't the ones you plan — they're the ones you stumble into while doing something else entirely.