A Druid-compatible engine for CI that boots in well under a second

Every pull request waits for the same docker-compose of JVMs to come up before the first integration test can run.

If you build a product on top of Apache Druid® — a connector, an analytics feature, an ingestion pipeline, anything that speaks Druid’s HTTP API — your integration tests need a live Druid endpoint to talk to. The standard answer is a docker-compose file that stands up a miniature cluster: Coordinator, Overlord, Broker, Router, Historical, MiddleManager, plus ZooKeeper and a metadata database. It works. It is also the slowest, flakiest step in the pipeline: each of those services is a JVM with its own startup, and the cluster is not queryable until ZooKeeper sessions are established, the metadata store is migrated, and service discovery has converged — a readiness gate that runs long before a single test assertion does, on every push, on every runner.

Consider a team — fictional, and deliberately generic — that ships a product against Druid’s native query and SQL endpoints. Their per-PR suite is a few hundred HTTP-level tests. The tests themselves are fast; the wait for the compose stack to answer its first query is what dominates the job, and the occasional readiness-race flake is what dominates the retries.

This article shows a different shape for that inner loop: FerroDruid — a single Rust binary that reimplements the Druid API surface (REST, native query JSON, Druid SQL) — started as one CI step, with no JVM, no ZooKeeper, and no external metadata database, answering its first SQL query in well under a second.

(Disclosure: FerroDruid is a commercial product built by abyo software LLC. It is an independent implementation and is not affiliated with or endorsed by the Apache Software Foundation. The team described above is a fictional composite; this article describes a verified test environment, not a customer deployment.)

One binary, one CI step

FerroDruid’s single-binary mode is one process serving the endpoints a Druid-facing test suite actually hits: /druid/v2 (native query JSON), /druid/v2/sql (Druid SQL), INFORMATION_SCHEMA introspection, and the /status family including the /status/health readiness probe. There is nothing else to orchestrate — no coordinator election, no ZooKeeper ensemble, no metadata-store container. The executable itself is a ~29.5 MB release binary that depends only on the system C library, so “install” in CI means “download one file.”

The measured startup story, with its conditions attached: on an AMD Ryzen 9 9950X running a FerroDruid v1.5.0 release build in single-binary mode (--no-auth, loopback), the time from process launch to the first answered SQL query was ~10 ms, with a 5-sample range of 0.006–0.011 s. Your CI runner is slower than that machine — which is exactly why the claim worth carrying is the conservative one: it boots and answers SQL in well under a second, leaving orders-of-magnitude headroom for shared runners.

The verified run, end to end, is small enough to paste whole. Start the binary in the background, poll /druid/v2/sql until SELECT 1 answers, then hit both query surfaces the way a test suite would:

READY in .010572384s
  native endpoint HTTP 200
[{"c":3}]  SQL INFORMATION_SCHEMA ok
  RSS: 17 MB

That last line is the resident memory of the serving process on the runner — ~16–17 MB idle in this run. The authoritative footprint numbers are the committed W-6 bench: 12.9 MB idle, 50.1 MB warm with 1,000 segments loaded, 74.2 MB active at 654 QPS (each state measured with its conditions recorded). For CI the practical consequence is that the “database” step is invisible next to the runner’s own tooling — it will not fight your build for memory.

The GitHub Actions job

Here is the whole thing as a workflow job — download the binary, start it, gate on readiness, run your suite against http://localhost:8888:

name: druid-api-integration
on: [push, pull_request]

jobs:
  integration-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Start FerroDruid (one process — no JVM, no ZooKeeper, no metadata DB)
        run: |
          # Fetch the ferrodruid binary from your artifact store / release
          # bucket (or `docker run` the container image instead).
          ./ci/fetch-ferrodruid.sh ./ferrodruid
          chmod +x ./ferrodruid
          ./ferrodruid serve --mode single-binary \
              --data-dir "$RUNNER_TEMP/fd-data" \
              --bind 127.0.0.1 --port 8888 --no-auth \
              > "$RUNNER_TEMP/ferrodruid.log" 2>&1 &
          echo $! > "$RUNNER_TEMP/ferrodruid.pid"

      - name: Wait for readiness (first SQL answer)
        run: |
          for i in $(seq 1 200); do
            curl -sf http://127.0.0.1:8888/status/health >/dev/null 2>&1 &&
            curl -sf -H 'Content-Type: application/json' \
                 -d '{"query":"SELECT 1 AS one"}' \
                 http://127.0.0.1:8888/druid/v2/sql | grep -q '"one":1' && exit 0
            sleep 0.05
          done
          echo "FerroDruid did not become ready" >&2
          cat "$RUNNER_TEMP/ferrodruid.log" >&2
          exit 1

      - name: Run the Druid integration suite
        env:
          DRUID_SQL_URL: http://127.0.0.1:8888/druid/v2/sql
          DRUID_NATIVE_URL: http://127.0.0.1:8888/druid/v2
        run: |
          # Your existing tests, pointed at the endpoints above —
          # native JSON queries, Druid SQL, INFORMATION_SCHEMA introspection.
          make druid-integration-tests

      - name: Stop FerroDruid
        if: always()
        run: kill "$(cat "$RUNNER_TEMP/ferrodruid.pid")" 2>/dev/null || true

Notes on the shape:

• --no-auth is for loopback CI only. The process binds 127.0.0.1 and lives for the length of the job; do not run that flag on anything reachable.
• The readiness loop is belt-and-braces. /status/health is the Druid-shaped probe; the SELECT 1 gate is the stronger guarantee that the SQL path itself is serving. In the measured run both were true on the first poll.
• A fresh process starts empty — that is a feature in CI. Hermetic suites load their fixtures per run; inline/local native-batch ingestion tasks and attaching pre-built segment fixtures are both supported paths, so each job gets a deterministic datastore with no state bleeding between runs.
• Prefer containers? The same engine ships as a container image, so a services: block or a docker run step works too.

Honest constraints

A CI story stands or falls on what happens when a query is not supported, so the boundaries come first, stated plainly:

• Core Druid SQL parity is approximately 95%, and the remainder fails loud, never wrong. The design rule is fail-closed: an unsupported surface returns a clear, Druid-shaped error — never a silently different answer. Concretely: JOIN and CTE (WITH …) queries parse and plan but are not wired end-to-end, so the SQL endpoint returns an explicit HTTP 501 with a Druid-shaped error envelope; a small set of edge functions is rejected with a stated reason. For a test suite this is the failure mode you want — a red test with a readable error, not a green test on a wrong result.
• Green against FerroDruid is not certification against Apache Druid. If your product’s production target is a Druid cluster, keep a full-Druid compose job as the release gate (nightly, or on the main branch) and use FerroDruid for the fast per-PR inner loop. The point of this article is to move the frequent runs off the slow stack, not to delete the authoritative one.
• HTTP only. The native JSON and SQL HTTP endpoints are the supported surface; suites that reach Druid through the JDBC/Avatica driver will not connect.
• The measured numbers carry their conditions. ~10 ms boot-to-first-SQL was measured on an AMD Ryzen 9 9950X with a release build in single-binary mode; shared CI runners will be slower. The claim to plan around is “well under a second,” not the milliseconds figure.
• License: FerroDruid is source-available under the Business Source License 1.1 (BUSL-1.1); each release converts to Apache-2.0 four years after its public release. It is not open source, and this article does not claim otherwise.

Call to action

If your pipeline spends longer standing up a Druid compose stack than running the tests against it, this is a cheap experiment: one binary, one job, your existing suite pointed at http://localhost:8888/druid/v2/sql — and the fail-loud errors will tell you exactly which of your tests touch an unsupported surface.

→ FerroDruid on AWS Marketplace (AMI) · FerroDruid Container for Amazon EKS · Product page

---

Apache Druid®, Apache Superset®, Apache Kafka®, and Apache® are registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. FerroDruid is an independent implementation developed by abyo software LLC and is not affiliated with, endorsed by, or sponsored by the Apache Software Foundation. Amazon Web Services, AWS, AWS Marketplace, Amazon EKS, and Amazon EC2 are trademarks of Amazon.com, Inc. or its affiliates. All other product names are trademarks of their respective owners. No compatibility claim in this article implies certification by any trademark holder.