Observability

FerroAir

Apache Airflow 3.x-compatible orchestrator

Single Rust binary, Airflow 3.x-compatible AWS service it replaces: Self-managed Apache Airflow

A self-managed workflow orchestrator you run on your own Amazon EC2 instances: a single static Rust binary that implements an Apache Airflow 3.x scheduler, webserver, REST API v2, triggerer, and DAG processor. It is wire-compatible with the Apache Airflow 3.0 / 3.1 / 3.2 REST API v2 surface and the AIP-72 task-execution interface, and task bodies keep executing on CPython via airflow.sdk.

FerroAir runs Apache Airflow 3.x workloads on your own EC2 (Apache Airflow is an open-source project, not an AWS service). It serves a genuine web UI (a Leptos single-page app, no Node runtime) on port 8080 (DAG list, graph, grid, gantt, calendar, and task-log views), a metrics & admin port 9080 with Prometheus metrics and health / live / ready probes, and AIP-72 on port 8081. It syncs DAGs from Amazon S3, stores metadata in PostgreSQL or SQLite, and runs an HA scheduler with leader election. The scheduler, webserver, triggerer, and DAG processor are unified in one static binary under a hardened systemd unit — instead of a multi-process Python deployment.

The problem

Apache Airflow is an excellent open-source workflow orchestrator (it is not an AWS service), but the usual deployment runs the scheduler, webserver, triggerer, and DAG processor as several Python processes, which is operationally heavy to manage. Teams end up assembling and maintaining a multi-process stack — metadata DB, broker, and workers — by hand. The need is to keep Airflow compatibility while making the orchestration layer simpler to run.

How it works

1
Runs as one Rust binary

FerroAir implements the Apache Airflow 3.x scheduler, webserver, REST API v2, triggerer, and DAG processor in a single static Rust binary, run unified under a hardened systemd unit. It runs on your own Amazon EC2 instances instead of a multi-process Python deployment.
2
Tasks still run on CPython

FerroAir owns the scheduler tick loop, DAG-folder watcher, REST API, metadata access, leader election, and executor dispatch in Rust. Task bodies keep executing on CPython through the upstream airflow.sdk over the Apache Airflow AIP-72 task-execution interface.
3
Wires into your existing state

DAGs sync from an Amazon S3 bucket (or local disk) and metadata is stored in PostgreSQL or SQLite. The web UI is served on port 8080, Prometheus metrics and probes on port 9080, and AIP-72 on port 8081.

Highlights

Airflow 3.x-compatible: REST API v2 + AIP-72; task bodies keep running on CPython via airflow.sdk.

One static Rust binary (scheduler + webserver + triggerer + DAG processor) instead of a multi-process Python deployment; a real Leptos web UI (no Node).

DAGs sync from S3, metadata in PostgreSQL / SQLite, HA scheduler with leader election; CloudFormation Quick Start included.

What's included

Amazon Linux 2023 AMI (x86_64, runs on c7i / m7i class instances)
A single static Rust binary implementing the Apache Airflow 3.x scheduler, webserver, REST API v2, triggerer, and DAG processor (wire-compatible with the Apache Airflow 3.0/3.1/3.2 REST API v2 surface and the AIP-72 task-execution interface)
A genuine web UI built as a Leptos single-page application (no Node runtime, port 8080) with DAG list, graph, grid, gantt, calendar, and task-log views
A dedicated metrics and admin port 9080 (Prometheus metrics plus health, live, and ready probes) and the AIP-72 port 8081 for the Python worker
Amazon S3 DAG sync, PostgreSQL or SQLite metadata (Apache Airflow 3 Alembic-compatible schema), and an HA scheduler with leader-election failover
An import tool for connections, variables, and pools from an existing managed Apache Airflow deployment, plus post-purchase docs with the runbook, compatibility matrix, and validation
No separate control plane, no telemetry home-call, and no license-key check (billed per instance per hour through your AWS bill, with an annual option)

Use cases

Running Apache Airflow workflows on your own Amazon EC2 as a single, operationally simpler binary

Teams migrating their orchestration layer from a managed Apache Airflow deployment by importing connections, variables, and pools

Existing workloads that need to keep working with their Apache Airflow REST API v2 clients

Running an orchestrator entirely inside your own VPC, with no separate control plane and no telemetry home-call

FAQ

How compatible is it with Apache Airflow?

FerroAir is wire-compatible with the Apache Airflow 3.0, 3.1, and 3.2 REST API v2 surface and the Apache Airflow AIP-72 task-execution interface. Because the REST API v2 surface is implemented, existing Apache Airflow REST API v2 clients are supported. See the product documentation for the full compatibility matrix.

Does my Python task code still run?

Yes. Task bodies keep executing on CPython through the upstream airflow.sdk over the Apache Airflow AIP-72 task-execution interface. The Rust binary owns the scheduler tick loop, DAG-folder watcher, REST API, metadata access, leader election, and executor dispatch — the task execution runtime itself stays CPython.

Is this a replacement for an AWS service?

Apache Airflow is an open-source project and is not an AWS service. FerroAir is a self-managed workflow orchestrator that runs those Apache Airflow 3.x workloads on your own Amazon EC2 instances. It ships as a normal Amazon Linux 2023 AMI that you run inside your own VPC.

Can I migrate from an existing Airflow deployment?

Yes. FerroAir can import connections, variables, and pools from an existing managed Apache Airflow deployment and sync your DAG folder. Metadata is stored in PostgreSQL or SQLite using an Apache Airflow 3 Alembic-compatible schema. The post-purchase documentation on the AMI walks through the import tool, the runbook, the compatibility matrix, and validation.

How does HA and secrets handling work?

The HA scheduler does failover via leader election (openraft); run several instances and leader election promotes a standby. Secrets resolve from environment, AWS Secrets Manager, GCP Secret Manager, Azure Key Vault, or Vault, and connections and variables are encrypted with Fernet. Authentication supports FAB auth, OAuth, OIDC, SAML, and LDAP.