← Blog

Migrating off Apache Druid without re-ingesting: reading S3 deep storage directly

Your data already sits in S3 in columnar form. A migration that starts by throwing that away deserves suspicion.

The re-ingestion tax

Every “migrate off Apache Druid®” story you can find follows the same shape: stand up the new system, replay all the source data into it, dual-run both stacks until the numbers agree, then decommission. The middle step is where migrations go to die, for reasons that have nothing to do with the destination technology:

  • The raw data may no longer exist. Druid clusters often ingest from Kafka topics whose retention is measured in days. The segments in deep storage are, for older intervals, the only remaining copy of that data in any form.
  • Re-ingestion is a full ETL project. The original ingestion specs encode accumulated transforms, rollup configs, and schema decisions — often written by people who have left. Reproducing them bit-for-bit in a new system’s ingestion layer can be weeks-to-months of work and its own source of silent discrepancies.
  • Dual-running doubles your infrastructure bill for the entire validation window, which for cautious teams can run to quarters, not weeks.

The irony is that Druid’s own architecture already solved the hard part. Deep storage — usually an S3 bucket — holds the authoritative, immutable, columnar form of every datasource. The cluster processes (Historicals) are just a serving cache over it. The data you would spend months re-ingesting is already sitting there, compressed and indexed.

So the obvious question: what if the replacement engine could just read the segments Druid already wrote?

How Druid actually stores your data

A quick tour, because the migration argument depends on it. An Apache Druid deployment has three data tiers:

  1. Deep storage (S3, HDFS, GCS, or local disk): the durable home of segments — immutable columnar files, one per (datasource, time interval, partition), each packaging the column data, dictionaries, and bitmap indexes for its slice of rows, laid out in Druid’s segment format.
  2. The metadata store (PostgreSQL/MySQL/Derby): a small relational database whose druid_segments table maps every published segment to its location in deep storage, along with its schema, version, and shard spec.
  3. The serving layer (Historicals + Brokers + the rest of the JVM fleet): pulls segments from deep storage onto local disk, memory-maps them, and answers queries.

Nothing in tier 3 owns any data. Kill the whole serving fleet and your data is intact; deep storage plus the metadata store is the database, in the same sense that a data lake’s object store is the lake. Druid’s own disaster recovery story rests on exactly this property.

Which means a migration engine has a much cheaper path available than re-ingestion: speak the segment format. Read druid_segments (or scan the bucket layout directly), fetch the same zipped segment files a Historical would fetch, decode the same columns, dictionaries, and bitmap indexes, and serve queries over them.

FerroDruid’s approach: same bucket, different engine

FerroDruid — a single Rust binary that reimplements the Druid API surface (REST, native query JSON, Druid SQL) without a JVM — reads Apache Druid’s v9 on-disk segment format, and does so along the exact path a migration needs: verified byte-for-byte against segments written by real Druid 31.0.2 and 35.0.1 on local disk, and end-to-end from a real AWS S3 deep-storage bucket (both demonstrations below). The intended migration shape is:

                    ┌─ (existing) Apache Druid cluster ─ decommission when ready
S3 deep storage ────┤
  (unchanged,       └─ FerroDruid, pointed at the same bucket
   still owned         and the same segment descriptors
   by you)
  • Deep storage stays where it is, in your account, unmodified. FerroDruid’s read path treats segments as the immutable artifacts they are.
  • There is no bulk conversion step and no second copy of the data to pay for.
  • Rollback is trivial by construction: the Druid cluster you are migrating away from can keep serving from the same unchanged bucket until you turn it off.

The columnar structures inside a segment (dictionary-encoded string columns, compressed numeric columns, bitmap indexes) map onto FerroDruid’s native segment reader, which implements decoding of the on-disk layout: the smoosh container files, index.drd, per-column descriptors, and the compression and dictionary encodings used by the default indexSpec.

Demonstration: reading Druid-written segments locally

This is not a format-spec reading exercise — the reader was built against observed bytes and then validated against segments produced by two real Apache Druid releases, 31.0.2 and 35.0.1. Both write on-disk format v9, and real Apache Druid through 36 still writes v9 on disk — so v9 is the format that matters for migration.

The validation method is deliberately strict: real Druid ingests a fixture dataset and publishes segments; FerroDruid reads those segment files off local disk with its own reader; then every column of every segment is compared against what Druid’s own SQL endpoint returns for the same data — a per-column SHA-256 deep match, not a “row counts look right” smoke test. The harness run covers 11/11 columns across 3/3 fixture segments on both Druid versions, over the default indexSpec encoding variants (LZ4, uncompressed, and 1- and 2-byte dictionary ordinals).

One honest boundary on that green result: it covers the default-indexSpec encoding surface — LZ4, uncompressed, and 1- and 2-byte dictionary ordinals — and everything outside it (ZSTD, front-coded dictionaries, Concise bitmaps, nested/sketch columns, longEncoding: auto, multi-value dimensions) fails loudly rather than being guessed at, as detailed in the limitations section. The local run above is a byte-for-byte check on disk; the next section takes the same reader out to a real S3 bucket.

Demonstration: reading from a real S3 deep-storage bucket

The local check proves the reader is correct. A migration needs one more thing proven: that the same reader works when the segment lives in the same place your Druid cluster keeps it — a real S3 bucket, reached through the deep-storage path, not a file copied to local disk first.

That end-to-end leg is verified. A genuine Apache-Druid-31-written v9 segment was uploaded to a real AWS S3 bucket, pulled back down through FerroDruid’s S3 deep-storage client, and read with the native segment reader — and it read back with its row count and column set intact: 5 rows, 11 columns, on the far side.

That is the migration thesis in miniature: a genuine Druid-written v9 segment, living in real S3, read back by a different engine. The gated integration test (s3_druid_segment_read.rs) ships in the tree and runs against a bucket and a real Druid segment you point it at, so the leg is reproducible, not just asserted — within the default-indexSpec surface the reader supports (anything outside it fails loud, per the limitations below).

One scope note kept deliberately precise, because migration decisions turn on it: the S3 leg proves the object-store round-trip (a real Druid-31 segment survives S3 upload/download and reads back with the same row count and column set), while the local leg proves byte-level decode fidelity. The strict per-column SHA-256 deep match is the local-disk result (against Druid 31.0.2 and 35.0.1); the S3 result is a row-count-and-column-set round-trip on a Druid-31 segment, not a byte-level match. Together they cover “the bytes decode correctly” and “the object-store path round-trips the segment” — the two things that can independently break a read-in-place migration. (The reverse direction — writing segments a Druid cluster can read back — is a separate matter covered under limitations below.)

What to check before you try this on your cluster (known limitations)

Honesty section, and with segment formats it matters more than usual: Druid’s on-disk format has many releases’ worth of generational variance, and a reader that pretends otherwise will hurt you. FerroDruid’s segment reader fails closed with a descriptive error on structures it does not implement, rather than guessing. The current known boundaries:

  • Legacy NULL semantics (Druid ≤ 27). Older Druid versions wrote segments under “legacy” NULL handling (empty string and NULL conflated); Druid 28+ defaults to SQL-compatible NULL storage. Segments written under the legacy mode carry different column semantics, and NULL-bearing columns in general are a validation frontier: the mapping is implemented but not yet measured against a NULL-bearing fixture.
  • Concise bitmaps. Druid’s older bitmap index encoding (before Roaring became the default) is not implemented; segments from clusters that never migrated their indexSpec off Concise will be rejected explicitly.
  • Non-default compression and encodings. LZF- or ZSTD-compressed columns, longEncoding: auto, and front-coded dictionaries are not covered by the reader; LZ4 and uncompressed layouts under the default indexSpec are the supported surface.
  • Nested columns and sketches in segments. Druid 28+ nested (JSON) columns are unsupported (this is also a query-layer gap), and sketch-bearing columns are outside the measured read surface.
  • Multi-value dimensions are outside the measured segment-read surface, and their filter semantics have known edge cases at the query layer.
  • This is a one-way door (today). The reverse direction — FerroDruid writing segments that an Apache Druid cluster can load — is not implemented; FerroDruid’s writer emits its own layout. Plan your rollback as “keep the Druid cluster stopped but intact,” not “write back.”
  • JOIN/CTE queries fail closed (HTTP 501) — detailed in the companion Superset article’s limitations section, and applicable regardless of how the data got in.

The practical consequence: the further your cluster’s indexSpec and Druid version history sit from “defaults, reasonably recent,” the more of your segments will be rejected (loudly) rather than read — and a pre-migration inventory pass over a sample of your segments is the first thing to run.

What this means for a migration plan

  1. Inventory pass — sample segments across datasources and interval ages; classify by format generation / indexSpec / bitmap type; estimate the readable fraction before committing to anything.
  2. Read-only pilot — point FerroDruid at deep storage (or a copied prefix), replay your top-N dashboard queries, diff results against the live Druid cluster. No writes, no risk.
  3. Cutover with the cluster warm — move query traffic; keep the Druid cluster stopped-but-intact as the rollback path; new ingestion lands in FerroDruid.
  4. Decommission — when the validation window closes, the JVM fleet and its metadata store go away; the bucket stays.

→ FerroDruid on AWS Marketplace (AMI) · FerroDruid Container for Amazon EKS · Product page

Apache Druid®, Apache Superset®, Apache Kafka®, and Apache® are registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. FerroDruid is an independent implementation developed by abyo software LLC and is not affiliated with, endorsed by, or sponsored by the Apache Software Foundation. Amazon Web Services, AWS, AWS Marketplace, Amazon S3, Amazon EKS, and Amazon EC2 are trademarks of Amazon.com, Inc. or its affiliates. All other product names are trademarks of their respective owners. No compatibility claim in this article implies certification by any trademark holder.

Estimate your AWS savings with the free tool →