S4 AccessLake
Every log to S3 + Iceberg, only the signal downstream
When does this pay back?
Payback depends on the cost line and replacement scope
This product's payback depends on your current AWS spend, replacement scope, and runtime environment. Use the bill calculator to identify matching cost lines and a rough savings estimate.
EC2 AMI of an AWS access-log lakehouse gateway that sits between your log sources and CloudWatch Logs / OpenSearch / SIEM. It archives every record full-fidelity to your own S3 as zstd + Apache Parquet + Apache Iceberg, and forwards only the signal downstream — cutting ingest volume 50–90% without dropping a single record.
S4 AccessLake sits between your AWS log sources (S3 server access logs, AWS WAF, CloudFront, ALB, NLB, VPC Flow Logs, Network Firewall, Route 53 Resolver) and the high-priced destinations they normally flow into (CloudWatch Logs, OpenSearch, SIEM). It archives the full-fidelity log stream as zstd + Apache Parquet + Apache Iceberg on your own S3 buckets, then forwards only the signals you actually need — critical events, top-N noisy IPs, aggregated counts, sampled traces. The result: 50–90% less CloudWatch / SIEM ingest GB while keeping every record for audit, incident response and forensics. A built-in cost dashboard quantifies your savings in real time, a policy DSL (dry-run rules + preflight lint validation) controls what flows where, and Merkle-anchored attestation lets third parties verify offline that the archive is complete.
The problem
Access logs from WAF, CloudFront, ALB, VPC Flow and friends make ingest pricing the dominant line item once you send everything to CloudWatch Logs or a SIEM (Splunk, Datadog, OpenSearch). But trimming volume with sampling or exclusions is how the one line you need during an audit or incident goes missing. The problem is structural: keep every record cheaply, and send only the signal to the expensive downstream.
How it works
- 1
Archive everything to S3 + Iceberg
Eight AWS access-log formats (S3 server access, WAF, CloudFront, ALB, NLB, VPC Flow, Network Firewall, Route 53 Resolver) arrive via SQS, Firehose or direct HTTP, are made durable in a WAL, then written to your own S3 bucket as a zstd raw archive. A normalized Parquet lake (Hive-partitioned, optionally Apache Iceberg on S3 Tables) is built as a derived view, queryable straight from Athena.
- 2
Forward only the signal via the policy DSL
You declare in a policy DSL which events go on to CloudWatch Logs, OpenSearch or your SIEM: matched critical events, top-N noisy IPs, aggregated counts, samples. New rules can run as dry-run first — hits are counted but nothing forwards — before promotion. Configs are lint-validated by the boot preflight, so a broken policy never starts serving.
- 3
Savings visibility and auditability
The built-in dashboard (loopback admin API, reached via SSM port forwarding) continuously totals the GB and dollars you no longer send downstream. A daily Merkle attestation manifest records archive completeness, verifiable offline by third-party auditors.
Highlights
50–90% reduction in CloudWatch Logs / SIEM ingest GB while keeping full-fidelity access logs on S3 + Apache Iceberg (zstd + Parquet).
Policy DSL with dry-run rules and preflight lint validation; Merkle-anchored raw-archive attestation for third-party audit.
Usage-priced Marketplace AMI with a 14-day free trial: per-normalized-GB metered via AWS Marketplace Metering Service; no license keys, IMDSv2 enforced, SSE-KMS by default.
What's included
- 8 log parsers: S3 server access, AWS WAF, CloudFront, ALB, NLB, VPC Flow, Network Firewall, Route 53 Resolver
- 7 forwarders including CloudWatch Logs / OpenSearch / SIEM targets, with retry control and an S3 DLQ
- zstd raw archive + normalized Parquet lake (Hive-partitioned; optionally Apache Iceberg on S3 Tables)
- Policy DSL with dry-run rules and preflight lint validation
- Savings dashboard + Prometheus-style /metrics (admin API is loopback-only; SSM port forwarding recommended)
- Daily Merkle attestation (optionally anchored to a second-region bucket)
- CloudFormation / Terraform quick starts, hardened systemd unit, IMDSv2 enforced, SSE-KMS by default
Use cases
Cut CloudWatch Logs ingest: everything lands on S3, only alert-worthy signal continues to CloudWatch
SIEM ingest tiering (Splunk / Datadog / OpenSearch): full fidelity in the lake, only curated events into the SIEM
Audit-regulated operators (FinTech, SaaS, EC): full retention plus forensics, queryable any time via Iceberg / Athena
Aggregate noisy IPs and attack patterns out of high-volume WAF / CloudFront logs before they hit downstream
FAQ
Which log sources are supported?
Eight sources: S3 server access logs, AWS WAF, CloudFront, ALB, NLB, VPC Flow Logs, Network Firewall, and Route 53 Resolver. Ingest paths are SQS (S3 events), a Firehose HTTP endpoint, and authenticated direct HTTP.
How is it priced?
Usage-priced: $0.010 per normalized GB ingested, metered via the AWS Marketplace Metering Service (there is no hourly software charge). You additionally pay the EC2 instance cost in your own account. A 14-day free trial is included, and Private Offers can customize per-GB rates and annual commitments.
Is x86_64 available?
Yes. This page covers the arm64 (Graviton) listing; x86_64 (t3 / m7i / c7i / r7i) ships as a separate companion listing with the same per-GB usage pricing.
Where does my data live?
Entirely in S3 buckets in your own AWS account. SSE-KMS by default, and the archive region is enforced against an allow-list. Telemetry is opt-in only; no data leaves your account.
How do I reach the dashboard?
The admin API and dashboard bind to loopback only (127.0.0.1:8080). You reach them via SSM Session Manager port forwarding — no inbound security-group rule required.
Pricing model
Usage-priced: $0.010 per normalized GB ingested (metered via AWS Marketplace Metering Service) + your own EC2 cost. 14-day free trial, no license keys. Private Offers customize per-GB rates and annual commitments. x86_64 ships as a companion listing.
Other S4 products
S4 — Squished S3
Transparent GPU S3-compression gateway
S4 Logs
Archive CloudWatch Logs to zstd S3
S4 Metrics
Govern CloudWatch metric cardinality