S4 — Squished S3
Transparent GPU S3-compression gateway
When does this pay back?
Example: $10,000/month in compressible S3 storage
At $10,000/month in compressible S3 storage spend, a 50–80% reduction is roughly $5,000–$8,000/mo — gross avoided usage charge before the S4 software fee, runtime compute, and workload differences.
Estimate your savings
Enter your relevant monthly spend or usage for a rough estimate — no bill upload needed.
Self-contained EC2 AMI of the S4 transparent S3-compression gateway with NVIDIA nvCOMP GPU codecs preinstalled. Launch on a GPU instance (g4dn / g5 / g6), point your S3 clients at it, and cut S3 storage bytes 50–80% for compressible data — with zero application changes.
S4 is a drop-in, S3-compatible gateway that transparently compresses every object on the way to your bucket. The AMI bundles Amazon Linux 2023, the NVIDIA driver, the container runtime and the GPU build of S4 (nvCOMP Bitcomp / zstd / GDeflate) preinstalled as a systemd service — no CUDA, driver or nvCOMP setup required. On a GPU instance it routes integer and columnar data (Parquet, ORC, postings, time-series) to nvCOMP GPU codecs and text or log data to CPU zstd, per object. Already-compressed inputs pass through untouched.
The problem
Your S3 storage bill grows linearly with the bytes you keep, and most of those bytes — logs, JSON, Parquet/ORC — are highly compressible. Your applications write them uncompressed because changing how they store data isn't worth the engineering effort, so you pay to store bytes you don't actually need. The result is a bill that climbs every month for data that could be 3× smaller or more.
How it works
- 1
Launch the GPU AMI
Launch the S4 AMI — Amazon Linux 2023 with the NVIDIA driver and nvCOMP preinstalled and S4 running as a systemd service — on a g4dn, g5, g6, or g6e instance inside your own VPC.
- 2
Repoint your S3 clients
Change only the endpoint URL on your boto3 / aws-cli / Spark / Trino / DuckDB clients; SigV4 auth, multipart upload, and Range GET keep working unchanged.
- 3
Compress transparently in transit
On each PUT the dispatcher samples the payload and routes it to the best codec — CPU zstd for text/logs, GPU nvCOMP Bitcomp for columnar integers, passthrough for already-compressed data — and GET returns the original bytes.
Highlights
GPU compression preinstalled: NVIDIA driver + nvCOMP (Bitcomp / zstd / GDeflate) baked into the AMI. Launch on g4dn / g5 / g6 and S4 uses the GPU automatically for integer and columnar data.
50–80% fewer S3 storage bytes for compressible data, with measured per-bucket savings reporting (s4 savings) built in.
Zero application changes and no lock-in: S3 wire-compatible endpoint, and objects stay readable without the gateway via Apache-2.0 CLI / Python / fsspec tooling.
What's included
- An EC2 AMI (Amazon Linux 2023) with the NVIDIA driver and nvCOMP preinstalled and S4 running as a systemd service, billed per instance-hour on your regular AWS invoice.
- Supported GPU instance families: g4dn, g5, g6, and g6e.
- An S3 wire-compatible endpoint — SigV4 / SigV4a auth, multipart upload, Range GET, and SSE — usable as a drop-in by any S3 SDK or tool.
- Per-payload codec dispatch across CPU zstd, GPU nvCOMP Bitcomp / zstd / GDeflate, and passthrough — chosen automatically by entropy and magic-byte sampling, with already-compressed inputs passed through untouched.
- Savings reporting: s4 estimate projects savings on an existing bucket before you deploy, and the s4 savings ledger reports the storage bytes and dollars actually saved, with Prometheus metrics and a Grafana dashboard included.
- Lock-in-free read tooling: the Apache-2.0 s4-codec CLI, the s4-codec Python package, the s4fs fsspec adapter (pandas / pyarrow / DuckDB), and a WASM browser decoder all read objects without the gateway.
- Day-2 tooling: s4 migrate retro-compresses existing objects and s4 recompact re-compresses cold data, while Range GET stays fast via a sidecar frame index compatible with Parquet/ORC readers.
Use cases
Teams ingesting high volumes of logs or JSON (nginx access logs, application logs) whose S3 bill is dominated by highly compressible text.
Data lakes storing Parquet/ORC and columnar integer data (postings, time-series) that want a smaller footprint while keeping Range-GET analytics fast.
Organizations with a monthly S3 bill above roughly $3,000, or already running GPU instances, where storage savings comfortably cover the GPU compute cost.
ETL / ML pipelines that want compression without rewriting application code — just repoint the S3 endpoint.
Case studies & benchmarks
Elasticsearch frozen-tier storage compression
Measured end-to-end on a real Elasticsearch 9.4.2 cluster (ES → S4 → MinIO, 4M-doc log index). Compressing the frozen-tier snapshot repository at the default zstd-3 cut storage by 27% (standard) / 22% (LogsDB) — up to 33% with s4 recompact — while cold frozen analytics queries stayed within ±1 ms of direct and snapshots gained ~no wall-clock. Stacked with LogsDB it yields a repository 2.82× smaller than plain standard-default.
Read the full benchmarkOpenSearch searchable-snapshot storage compression
Measured end-to-end on a real OpenSearch 2.19 cluster (OpenSearch → S4 → MinIO, 4M-doc log index). Compressing the searchable-snapshot S3 repository with S4 cut storage by 28% (default) / 17% (best_compression / zstd / zstd_no_dict). Because OpenSearch's index.codec only compresses stored fields, S4 still saves ~17% on top of the native zstd codec by squeezing doc-values and postings. In this local run remote_snapshot search latency stayed within ~1.5 ms of direct (S4 equal or faster). Requires S4's --logical-etag flag for repository-s3 (added in this work).
Read the full benchmarkGrafana Loki chunk compression
Measured end-to-end on a real Grafana Loki 3.3.2 instance (Loki → S4 → MinIO, 4M log lines). Re-compressing Loki's snappy chunks with S4 zstd-3 cut bucket storage by 18.4% (zstd-9 19.3%). Stated honestly: switching Loki's own chunk_encoding to zstd saves 38% on new chunks — more than S4 — so S4's wedge is the forward-only immutable snappy backlog, not greenfield (complementary, not a replacement). Reads: a whole-chunk GET through S4 costs ~1.7 ms more (byte-identical; decompress cost, local run). Unlike OpenSearch, --logical-etag is not required for Loki (recommended for correct ETags).
Read the full benchmarkKafka tiered-storage compression
Measured end-to-end on a real Apache Kafka 3.9.1 cluster (KIP-405 tiered storage + the Aiven plugin → S4 → MinIO, 600k records/topic). Re-compressing the tiered log segments with S4 zstd-3 cut stored bytes by 74.7% for none (uncompressed) / 22.6% snappy / 20.6% lz4 / 0.0% zstd, by producer compression.type. Stated honestly: if the producer sends zstd it shrinks segments at the source (43.9 MB — the same floor S4 reaches over none, 42.0 MB), and S4 adds ~nothing over already-zstd segments. So S4's value is the segments of none/snappy/lz4 or can't-change-the-producer cases (complementary, not a replacement). No consistent S4 cold remote-fetch penalty (local run, single noisy samples). Unlike OpenSearch, --logical-etag is not required for Kafka (recommended for correct ETags).
Read the full benchmarkCold Parquet recompaction
Measured locally against MinIO (a 2,000,000-row ECS-style log Parquet, 13 columns). A new s4 parquet-recompact subcommand re-encodes cold data-lake Parquet column chunks to zstd and writes back a NATIVE Parquet — pyarrow / Spark / Trino / DuckDB read it directly, with no S4 in the read path. Over snappy (the data-lake default) it cut storage 36.6%, over uncompressed 51.7%, gzip 3.8%, already-zstd 0.0% (skipped). Every output is value-for-value identical (pyarrow table.equals). Stated honestly: setting the writer's codec to zstd reaches the same floor at the source (79.4 MB), so S4's wedge is the existing snappy/none/gzip backlog no one re-runs a writer job for (complementary, not a replacement). Each object is value-verified per row group (bounded memory) before the in-place overwrite, and unverifiable objects are never overwritten.
Read the full benchmarkFAQ
If I stop running S4, can I still read my data?
Yes. The compressed objects and their S4IX sidecars stay S3-native and remain listable and downloadable with stock aws-cli or boto3. To recover the original payload you use the Apache-2.0 s4-codec CLI, the s4-codec Python package, the s4fs fsspec adapter (pandas / pyarrow / DuckDB), or the WASM browser decoder — all open source, pure decode, no gateway runtime required.
Do I have to change my applications?
No. S4 speaks the S3 wire protocol — same SigV4 auth, same multipart, same Range GET, same SDK calls. You change only the endpoint URL your boto3 / aws-cli / Spark / Trino / DuckDB client points at; nothing else changes.
Which part of my bill actually gets cheaper?
Storage bytes only. S4 cuts the bytes stored in S3 by 50–80% for compressible data — for example, 1 TiB of nginx logs compresses to about 6.6 GiB. Request cost (PUT/GET), egress, and GPU compute are unchanged. Because of the GPU instance cost, S4 typically pays off above roughly a $1,000/month S3 bill (lower on spot instances, or if you already run GPUs).
Which GPU instances does the AMI support?
The EC2 AMI (Amazon Linux 2023, NVIDIA driver + nvCOMP preinstalled) runs on g4dn, g5, g6, and g6e instances, billed per instance-hour on your regular AWS invoice. There is no license key — metering is handled through AWS Marketplace.
Where does S4 run, and who can see my data?
You launch the AMI in your own AWS account and VPC, and it talks to your own S3 bucket — no data leaves your account, and there is no third-party service in the path. It supports TLS and server-side encryption (SSE).
Why it's cheaper
Assumes 100 TB stored in S3 Standard (us-east-1) on data that compresses ~3× (Parquet/ORC, JSON, etc.).
- S3 storage (100 TB)
- $2,300 / mo
- Monthly total
- $2,300 / mo
- S3 storage (34 TB compressed)
- $800 / mo
- S4 instance (g4dn.xlarge + software fee)
- $385 / mo
- Monthly total
- $1,185 / mo
Sizing the S4 instance by write load
| Write load | Recommended instance | S4 instance cost | Total at 100 TB stored |
|---|---|---|---|
| ~1 TB / day | g4dn.xlarge | $385 / mo | $1,185 / mo (−49%) |
| ~10 TB / day | g6.xlarge | $590 / mo | $1,390 / mo (−40%) |
| High-concurrency / 100 TB / day | g6.4xlarge | $965 / mo | $1,765 / mo (−23%) |
Illustrative example. Storage is priced from the published S3 Standard (us-east-1) rate ($0.023/GB-month) and EC2 on-demand pricing for g4dn / g6. The S4 software fee is billed hourly via AWS Marketplace, separate from the EC2 charge (a 1-year Savings Plan typically cuts the EC2 portion by 30–40%). Actual savings depend on your data's compressibility (Parquet, JSON ~3–5×; text logs ~4–10×) and read/write patterns.
Pricing model
Hourly software fee + the GPU EC2 you choose (g4dn / g5 / g6). Metered per instance type, annual option available, no license keys.
Other S4 products
S4 Logs
Archive CloudWatch Logs to zstd S3
S4 Metrics
Govern CloudWatch metric cardinality
S4 NAT
Cost-optimized NAT for Amazon VPC