Rehearsing a migration off a legacy Apache Druid cluster — without touching the original

The scariest sentence in a migration review is not “it will take six months.” It is “the numbers changed, and nobody can say why.”

If you run a 2019-era Apache Druid® + Apache Superset® analytics stack, you probably recognize this team (they are fictional, but the situation is not rare): a mid-size SaaS company whose analytics cluster was built years ago by an engineer who has since left. The dashboards still work. They are full of hyperUnique distinct-counts — Druid’s classic built-in sketch from before the DataSketches era — and the cluster runs on a Druid generation old enough that its legacy null handling (useDefaultValueForNull=true, the default through Druid 27) is baked into every stored segment: empty string is null, numeric null is zero.

The team would love to stop operating the JVM fleet, the ZooKeeper ensemble, and the metadata database. What stops them is not attachment to Druid. It is the fear that after any migration, a distinct-count on a board-level dashboard shifts by a few percent and there is no one left who can explain whether that is a bug, a semantics change, or reality.

This article is the antidote to that fear: a migration rehearsal you can run end-to-end on one machine, with free software, against a stand-in legacy cluster — and every number checked against Druid’s own answers before anything real is touched. The original cluster is never modified: FerroDruid reads the segments Druid already wrote to deep storage, as the immutable artifacts they are.

(Disclosure: FerroDruid is a commercial product built by abyo software LLC. It is an independent implementation and is not affiliated with or endorsed by the Apache Software Foundation. The team described above is fictional and generic; this article describes a verified test environment, not a customer deployment.)

The rehearsal, on one page

1. Stand up a faithful legacy stand-in:  real Apache Druid 27.0.0
   (legacy nulls BY DEFAULT) + S3 deep storage (MinIO, free/local),
   ingest a dataset with a hyperUnique metric and null-bearing columns.
2. Capture the oracle:                   Druid 27's own answers, SQL + native.
3. Boot FerroDruid:                      one Rust binary, no JVM/ZooKeeper.
4. Attach the segments straight from S3: no re-ingestion, no ETL.
5. Flip on legacy null mode:             --use-default-value-for-null.
6. Point the same Superset at it:        same dashboards, same numbers.
7. Check the scariest number first:      hyperUnique — down to the last bit.
8. Confirm it is not a one-way door:     FerroDruid also writes v9 Druid reads.

Everything below was actually executed (2026-07-21, AMD Ryzen 9 9950X, FerroDruid v1.5.0 release build), and the outputs shown are real captures. The whole rehearsal reproduces on free, local infrastructure; a re-run of the attach leg against genuine AWS S3 is covered near the end as an “also verified” note.

Step 1 — Reproduce the legacy environment

You cannot rehearse against production, so you rehearse against a faithful stand-in. Apache Druid 27.0.0 is the right stand-in for the whole legacy generation: it is the newest release where useDefaultValueForNull=true is still the default (SQL-compatible nulls only became the default in Druid 28), and — like every Druid of its era and earlier — it frames numeric columns on disk with the v1 column serdes (long/double/float) rather than the v2 serdes modern Druid writes. If an engine can read what Druid 27 writes under legacy nulls, it can read what your 2019 cluster wrote.

The stack is three containers: Druid 27 single-node (micro-quickstart layout), MinIO as the S3 deep-storage endpoint, and a one-shot bucket-setup job. The Druid S3 configuration that actually works against MinIO on Druid 27, in common.runtime.properties:

druid.extensions.loadList=["druid-s3-extensions","druid-datasketches","druid-kafka-indexing-service"]
druid.storage.type=s3
druid.storage.bucket=druid-deep-storage
druid.storage.baseKey=druid/segments
druid.storage.disableAcl=true          # MinIO has no AWS-style object ACLs
druid.s3.accessKey=minioadmin
druid.s3.secretKey=minioadmin
druid.s3.protocol=http
druid.s3.enablePathStyleAccess=true    # REQUIRED for MinIO
druid.s3.endpoint.url=http://minio:9000
druid.s3.endpoint.signingRegion=us-east-1

Three reproducibility gotchas we hit live, so you don’t have to: quickstart- style Druid images read configuration files, not druid_* environment variables — bind-mount the properties file over the micro-quickstart _common/common.runtime.properties; enablePathStyleAccess and disableAcl are both mandatory for MinIO; and value and day are Druid SQL reserved words, so quote the column (SUM("value")) and rename the alias.

The dataset is small on purpose — 20 rows over two UTC days, engineered so every interesting number has a known ground truth:

• user: 8 distinct users (u1…u8), split 5 on day 1 and 5 on day 2 with a deliberate overlap, so the union of 8 can only be computed by merging hyperUnique sketches across segment boundaries — the case a migration must get right.
• value (long): 4 rows are null in the source.
• page (string): 4 rows are the empty string, 2 rows are absent — under legacy nulls, all 6 collapse into one null bucket.
• Metrics: a count, a longSum of value, and uu — a hyperUnique of user, exactly the pre-DataSketches distinct-count the fictional team’s dashboards are full of.

Ingested via a native batch index_parallel task with segmentGranularity=DAY, this produces 2 segments, which Druid 27 pushes to MinIO. The observed S3 key layout — worth recording, because a migration tool must handle it as-is:

druid/segments/uc1_events/2024-01-01T00:00:00.000Z_2024-01-02T00:00:00.000Z/2026-07-21T11:22:29.534Z/0/index.zip
druid/segments/uc1_events/2024-01-02T00:00:00.000Z_2024-01-03T00:00:00.000Z/2026-07-21T11:22:29.534Z/0/index.zip

Two facts about that layout matter: the interval and version directories keep their colons intact (Druid 27’s S3 pusher writes raw ISO timestamps into object keys), and no descriptor.json is written to deep storage — only index.zip; segment metadata lives in Druid’s metadata store (loadSpec type=s3_zip). A reader must parse identity out of the key path itself.

Finally, the oracle: Druid 27’s own answers, captured over both the SQL and native endpoints. Highlights (full capture in the evidence file):

Query (against Druid 27)	Druid 27’s answer	Known truth
SELECT COUNT(*)	20	20
SELECT APPROX_COUNT_DISTINCT(uu)	8	8
native hyperUnique(uu) merge, raw estimate	8.015665809687173	~8
GROUP BY country → APPROX_COUNT_DISTINCT(uu)	US=5, JP=3, DE=3	5/3/3
SELECT SUM("value")	955	nulls contribute 0
SELECT COUNT(page)	14	'' counts as null
COUNT(*) WHERE page IS NULL	6	4 empty + 2 absent
COUNT(*) WHERE page = ''	6	same 6 rows: '' == null
GROUP BY page	""=6, /home=6, /about=4, /=2, /contact=2	null+'' merged

That last block is legacy null handling in its purest form: page IS NULL and page = '' return the same six rows, and GROUP BY has a single merged empty/null bucket. A SQL-compatible engine would split them. This is exactly the class of silent drift the rehearsal exists to catch.

Step 2 — Boot the replacement engine

FerroDruid is a single Rust binary (29.5 MB, release build) that speaks the Druid API surface: /druid/v2/sql, the native query JSON endpoints, and INFORMATION_SCHEMA. Booting the single-binary mode on the test machine (AMD Ryzen 9 9950X, v1.5.0 release, --no-auth, empty data directory), measured as boot-to-first-successful-SQL-answer, five samples:

sample 1: boot=0.008s  rss=15.9MB
sample 2: boot=0.006s  rss=15.8MB
sample 3: boot=0.007s  rss=15.8MB
sample 4: boot=0.007s  rss=15.7MB
sample 5: boot=0.007s  rss=16.2MB

It boots and answers its first SQL query in well under a second, idling in the mid-teens of megabytes. The authoritative, committed memory-footprint bench (repo evidence, same product) puts it at 12.9 MB idle, 50.1 MB warm with 1,000 segments resident, and 74.2 MB active at 654 queries/second — the fresh re-measure above is consistent with it. For a rehearsal, the practical meaning is: standing up the candidate engine costs you a laptop process, not a capacity-planning meeting.

Step 3 — Attach the Druid segments straight from S3

No re-ingestion, no ETL, no touching the source. ferrodruid-migrate has a dry-run assess and a committing attach, both of which take the deep- storage root directly — including an s3:// URL (for MinIO and other S3-compatible stores, set AWS_ENDPOINT and AWS_ALLOW_HTTP):

AWS_ACCESS_KEY_ID=minioadmin AWS_SECRET_ACCESS_KEY=minioadmin \
AWS_REGION=us-east-1 AWS_ENDPOINT=http://localhost:29000 AWS_ALLOW_HTTP=true \
  ferrodruid-migrate attach \
    --deep-storage s3://druid-deep-storage/druid/segments \
    --datasource uc1_events \
    --ferro-deep-storage ~/ferro-demo/deep-storage \
    --metadata-uri ~/ferro-demo/metadata/ferrodruid.db

Before showing the success, it is worth showing what failure looks like, because with segment formats the failure mode is the product. During v1.5.0 development, an earlier build ran this exact attach against these exact segments and refused them — loudly:

FAILED  -  uc1_events_2024-01-01T00:00:00.000Z_2024-01-02T00:00:00.000Z_..._0
   reason: not readable by the v9 reader: segment error: v9 segment: failed to
   decode column `__time`: segment error: druid-native: unsupported column
   part type `long` (valueType LONG)

That long (not longV2) is the legacy v1 numeric column serde — the on-disk framing older Druid uses for __time and every numeric metric. This is precisely what makes “reading a genuinely old cluster” different from “reading a demo segment written by a current Druid.” The v1.5.0 release reader decodes the v1 long/double/float serdes, reverse-engineered clean-room from the observed bytes of real Druid 27 segments and verified byte-exact against Druid 27’s own dump-segment tool (longs including the auto TABLE/DELTA sub-encodings, doubles bit-exact including π, floats). Note the design stance in that error message, though: a structure the reader does not implement is a descriptive rejection, never a guess.

With the release build, the same command:

staged 2 object(s) (3093 bytes) from s3://druid-deep-storage/druid/segments
  into a private local staging dir
ATTACHED  10 rows  uc1_events_2024-01-01T00:00:00.000Z_2024-01-02T00:00:00.000Z_2026-07-21T11:22:29.534Z_0
ATTACHED  10 rows  uc1_events_2024-01-02T00:00:00.000Z_2024-01-03T00:00:00.000Z_2026-07-21T11:22:29.534Z_0
Summary: 2 attached, 0 would attach (dry run), 0 skipped (already attached),
  0 failed, 0 filtered out (--datasource), 2 artifact(s) processed.
Next step: restart the FerroDruid instance (`ferrodruid serve`) — its startup
  bootstrap reload downloads, hash-verifies, and loads every attached
  segment; it is then query-visible.

2 attached, 0 failed, 20 rows — and the source bucket was only ever read. Being precise about the mechanics, because migration decisions turn on them: attach stages the S3 objects into a private local temp directory, gates each one on the real v9 reader, then transcodes it into the target FerroDruid’s deep storage, which in this release is local to the target (the source stays in S3; the target copy does not). It is an offline importer: segments become queryable at the next startup’s bootstrap reload, which is exactly what the serve log then shows:

FerroDruid v1.5.0 — single-binary mode
WARN … useDefaultValueForNull=true (LEGACY null mode): null strings == '' and
       null numerics == 0, matching Apache Druid <= 27 defaults. …
INFO … metadata store initialized backend="sqlite"
INFO … deep storage initialized (local filesystem)
INFO … bootstrap reload complete reloaded=2
INFO … FerroDruid v1.5.0 listening on 127.0.0.1:18945

Step 4 — Turn on legacy null semantics

The serve line above already gave it away: the process was started with --use-default-value-for-null, FerroDruid’s opt-in legacy null mode (named after Druid’s own property). Under the flag, string null ≡ '' and numeric null → 0, matching Druid ≤ 27; with the flag off (the default), FerroDruid keeps modern ANSI null handling, byte-for-byte unchanged.

ferrodruid serve --mode single-binary \
  --data-dir ~/ferro-demo \
  --use-default-value-for-null

Why this matters for legacy-written segments specifically: Druid 27 already coerced those 4 null value rows to literal 0 and those 6 empty/absent page cells to '' at ingest time. The distinction is gone from the bytes. An engine reading them under ANSI semantics would truthfully report page IS NULL → 0 — truthful about the bytes, and a silent behavioral break from what every existing dashboard shows. (FerroDruid flags this exact trap independently: the boot log’s NULL GENERATION UNCONFIRMED warning fires when a column’s values are legacy-null-consistent but carry no null markers.)

With the flag on, the rehearsal’s null-semantics check against the Druid 27 oracle, on the attached-from-S3 segments:

Query	FerroDruid (legacy null mode)	Druid 27 oracle
SELECT COUNT(*)	20	20
SELECT SUM("value") (null→0)	955	955
SELECT COUNT(page) (''≡null)	14	14
COUNT(*) WHERE page IS NULL	6	6
GROUP BY page	""=6, /home=6, /about=4, /=2, /contact=2	identical

Every cell identical, including the single merged null+'' bucket in the GROUP BY. Behind this small table sits the committed v1.5.0 verification battery: 52 captured oracle answer files per engine against real Druid 27.0.0 (legacy) and 31.0.2 (ANSI), asserted by dedicated end-to-end tests.

Step 5 — Point Superset at it: same dashboards, same numbers

The fictional team’s real question is not “does the SQL endpoint work” — it is “do the dashboards my executives look at render the same numbers.” So the rehearsal drives a real Apache Superset 4.1.4 (pydruid 0.6.9), connected the same way it would connect to a Druid Broker:

druid://<ferrodruid-host>:18950/druid/v2/sql/

Four charts were built in the live Superset UI over the attached, legacy-mode datasource — chosen to be exactly the shapes this migration fears — plus a dashboard assembling them. Every rendered number was verified against the Druid 27 oracle, and each chart’s SQL was additionally POSTed to the still-running Druid 27 container, with byte-identical responses.

_{Distinct users by country — APPROX_COUNT_DISTINCT(uu) over the Druid-27-written hyperUnique column: US 5, DE 3, JP 3. Oracle: identical.}

_{Total distinct users — the cross-segment sketch merge: day 1 has 5 users, day 2 has 5, and the correct union is 8, which requires merging hyperUnique sketches across both attached segments. Rendered: 8. Oracle: 8.}

_{SUM(value) by day — 2024-01-01 → 195, 2024-01-02 → 760 (Σ = 955, with the four legacy null rows contributing 0). Cross-checked against live Druid 27: identical rows.}

_{Rows by page — the legacy-null payoff in a chart: the '' bucket holds 6 rows (4 empty + 2 absent, collapsed), then /home 6, /about 4, / 2, /contact 2. Oracle: identical.}

_{The assembled dashboard — all four tiles rendering live against FerroDruid over the stock pydruid connector.}

One reproducibility note that will save you an hour: Superset’s pydruid reflection records the synced __time column as LONG, so set its type to TIMESTAMP once in the dataset editor before building time-series charts. This is a Superset-side reflection quirk — on the wire, both the SQL results and INFORMATION_SCHEMA report __time as TIMESTAMP — and it applies when connecting Superset to either engine via pydruid.

The answer-check: hyperUnique, down to the last bit

The rounded SQL integers matching is necessary but not fully reassuring — an approximate counter can round two different estimates to the same integer. So the rehearsal checks the strongest witness available: the raw native double that the merged hyperUnique sketch estimates, before any rounding.

Druid 27’s native groupBy over hyperUnique(uu), merging the stored sketches across both segments, answers 8.015665809687173. FerroDruid, reading those same Druid-written sketch bytes from the attached segments and running its own clean-room decoder and estimator:

[{"timestamp":"1970-01-01T00:00:00.000Z","result":{"uu_est":8.015665809687173}}]

The same double, bit for bit — not “approximately 8,” but the identical IEEE-754 value Druid computes, produced by an independent implementation reading Druid’s bytes. The committed v1.5.0 test suite pins this class of result with to_bits() equality against Druid 31.0.2-captured oracles across single-user, rollup-folded, dense (~700 distinct per sketch), and 6-segment multi-shard merge fixtures, with SQL rounding pinned to Java Math.round behavior. For the fictional team, this is the sentence that unblocks the migration review: the distinct-counts are not “close”; they are the same numbers. (The precise scope and the honest caveats on this claim — it has them — are in the constraints section below.)

Also verified: the same attach against real AWS S3

Everything above runs on free, local infrastructure — that is the point of a rehearsal. But “S3-compatible” claims deserve one real-S3 proof, so the spine was re-shot against genuine AWS S3 (us-east-1): the same Druid 27 ingested the same 20-row dataset into a scratch bucket, and the observed real-S3 key layout matched MinIO exactly — interval colons intact in the object keys, no descriptor.json in deep storage. The same ferrodruid-migrate attach --deep-storage s3://… (real AWS credentials, no endpoint override) attached 2/2 with 0 failures, and the queries matched the oracle again: COUNT(*)=20, SUM("value")=955, APPROX_COUNT_DISTINCT(uu)=8, COUNT(page)=14, page IS NULL=6. The scratch bucket and containers were torn down in-run (head-bucket 404 verified). The MinIO rehearsal is faithful to real S3.

Reversibility: a rehearsal is not a one-way door

An evaluation you cannot back out of is not an evaluation. Two properties keep this one reversible:

1. The source is never modified. Attach reads deep storage; the Druid cluster and its bucket remain intact throughout. The rollback plan during a pilot is simply “keep pointing the dashboards at Druid.”
2. FerroDruid can also write segments Druid reads back. The native v9 writer emits real Druid v9 segments, verified by having Apache Druid 31’s own dump-segment tool read them back with a per-column SHA-256 deep match against the source data — byte-identical to Druid-written fixtures, including the UTF-16-order string dictionaries Druid expects. The honest scope in this release: LONG columns and single-value STRING dimensions (double/float/nullable/multi-value/compressed writing is a documented follow-on). Not yet a full reverse migration — but the door swings both ways at the format level, and that changes the risk calculus of stepping through it.

Honest constraints

A rehearsal article earns its keep by stating exactly where the edges are. FerroDruid’s design rule is fail loud, never silently wrong — every boundary below is an explicit error, not a quietly different number.

• Scale of this evidence. The rehearsal dataset is 20 rows across 2 segments. That is deliberate — it makes every number hand-checkable and the sketch merge provable — but it demonstrates wire and semantic fidelity, not scale. Run your own segments and your own dashboard mix before concluding anything about performance.
• Sketch coverage. hyperUnique columns decode (this article) and thetaSketch columns decode (verified against real Druid 31); DataSketches HLLSketch and quantilesDoublesSketch columns do NOT decode — they are rejected loudly, or dropped per-column under the opt-in --allow-unreadable-columns lenient attach.
• hyperUnique bit-exactness has a measured scope. Every captured oracle fixture falls in the HLL linear-counting regime; the high-cardinality raw-HLL estimator branch (≳ tens of thousands of distinct per folded sketch) is implemented from the public HLL literature and is not oracle-verified. Bit-exactness of the unrounded native double is also libm-dependent (measured on glibc; rounded SQL integers are safe).
• Legacy null mode covers the oracle-pinned surfaces. Aggregates, ''≡null filters, group keys/ordering, distinct counts, and empty-set sentinels are verified cell-for-cell against real Druid 27. Surfaces without a captured oracle — string bound/like/regex filters over the merged value, first/last, window functions, joins — deliberately keep ANSI behavior under the flag: a documented refusal to guess, not a silent divergence. The mode is startup-latched (restart to flip, like Druid) and disables the vectorized fast paths — it is a migration-compatibility mode, not a performance mode.
• Attach mechanics. S3 sources are staged to a private local temp directory; the target FerroDruid’s deep storage is local in this release; single-binary scope; offline importer (segments become queryable at the next startup’s bootstrap reload). Path-derived identity assumes Druid’s conventional <datasource>/<interval>/<version>/<partition> key layout (an unconventional layout is a loud per-segment skip; the separate metadata-DB importer is layout-immune).
• Segment encodings. The v9 reader covers the default-indexSpec surface plus the legacy v1 numeric serdes shown here; Concise bitmaps, ZSTD/LZF, front-coded dictionaries, and nested (JSON) columns fail loud. A pre-migration ferrodruid-migrate assess over a sample of your real segments is the honest first step.
• SQL surface. Approximately 95% core Druid SQL parity; unsupported surfaces fail loud — notably JOIN and CTE execution returns a Druid-shaped HTTP 501 rather than executing. Let the errors tell you which of your queries touch the boundary.
• Write-back scope. v9 write-back is LONG + single-value STRING this release, as stated above.
• This walkthrough’s posture is not production’s. --no-auth and the public-bind override are for the local rehearsal only.
• License. FerroDruid is source-available under the Business Source License 1.1, with each release converting to Apache-2.0 four years after its release date. That is not the same thing as open source, and we don’t call it that.

Call to action

The rehearsal above is the pilot plan: stand up the free stand-in, run assess over a sample of your real segments, attach a datasource, flip on legacy null mode, point a staging Superset at it, and diff your own dashboards against your own cluster — before anyone commits to anything. The original cluster is never touched.

→ FerroDruid on AWS Marketplace (AMI) · FerroDruid Container for Amazon EKS · Product page

---

Apache Druid®, Apache Superset®, Apache Kafka®, and Apache® are registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. FerroDruid is an independent implementation developed by abyo software LLC and is not affiliated with, endorsed by, or sponsored by the Apache Software Foundation. Amazon Web Services, AWS, AWS Marketplace, Amazon S3, Amazon EKS, and Amazon EC2 are trademarks of Amazon.com, Inc. or its affiliates. All other product names are trademarks of their respective owners. No compatibility claim in this article implies certification by any trademark holder.