Apache Airflow

## Editorial notes

Apache Airflow is useful when Claude-adjacent teams need mature workflow-as-code scheduling for data pipelines, report generation, batch jobs, model evaluation runs, data quality checks, warehouse tasks, provider integrations, and operational monitoring. It gives teams a shared DAG model, scheduler, worker execution, UI, command-line tools, provider ecosystem, task logs, and production deployment patterns.

This entry is a dedicated tool listing. Existing repository content mentions Apache Airflow inside a data pipeline engineering agent and a data engineering collection, but those are contextual references rather than a standalone source-backed Airflow entry. It is distinct from Dagster, which centers on data assets, lineage, and asset-oriented orchestration. Airflow centers on DAG-based workflow authoring, scheduling, execution, and monitoring.

## Source notes

- The official repository describes Apache Airflow as a platform to programmatically author, schedule, and monitor workflows.
- The README says workflows defined as code become more maintainable, versionable, testable, and collaborative.
- The README says Airflow authors workflows as DAGs, the scheduler executes tasks on workers while following dependencies, and the UI visualizes and troubleshoots production pipelines.
- The README says Airflow works best with mostly static and slowly changing workflows, encourages idempotent tasks, and recommends delegating high-volume data-intensive work to external specialized services.
- The README says Airflow is not a streaming solution, though it is often used to process real-time data in batches.
- The README documents dynamic pipelines, extensibility through operators, flexibility through Jinja templating, official source releases, convenience packages, Docker images, and Helm charts.
- The README documents tested Python, platform, Kubernetes, PostgreSQL, MySQL, and SQLite versions for current and stable releases, and says production execution should use Linux-based distributions.
- The README says only `pip` installation is currently officially supported and recommends constraint files for repeatable installs.
- The production deployment docs warn that SQLite is for testing only and recommend an external metadata database for production.
- The security model docs describe deployment managers, authenticated UI users, DAG authors, admin users, connection configuration users, and the trust required because DAG author code can execute arbitrary Python.
- The secrets docs cover variables, connections, encryption at rest, secrets backends, and masking sensitive values in logs, UI, and rendered fields.
- The XCom docs describe XComs as per-task-instance communication intended for small values, with object storage recommended for larger data.
- The task logging docs describe local and remote task logs, remote backends, log serving from workers or triggerer, and configuration options.
- The repository is `apache/airflow`, is Apache-2.0 licensed, and is active.

## Duplicate check

Checked current `content/tools/`, `content/mcp/`, agents, hooks, rules, skills, commands, guides, collections, open pull requests, live issue state, and repository-wide content for `Apache Airflow`, `Airflow`, `apache/airflow`, `github.com/apache/airflow`, `airflow.apache.org`, `DAG orchestration`, and `workflow orchestration`. Existing references appear only inside `content/agents/data-pipeline-engineering-agent.mdx` and `content/collections/data-engineering-suite.mdx`; no dedicated Apache Airflow tools entry, source URL duplicate, target file, issue duplicate, or open duplicate PR was found.

## Disclosure

Editorial listing. No paid placement or affiliate link is used. Apache Airflow is Apache-2.0 open-source software; providers, executors, databases, Kubernetes clusters, cloud services, logging backends, secrets managers, and downstream systems may have separate licenses, billing, terms, privacy obligations, and access controls.