Airflow alternatives

Most popular alternatives to Apache Airflow are the following:

• Luigi
• Apache NIFI
• AWS Step Functions
• Pachyderm
• Kedro
• Prefect
• Dagster

Streaming systems require a refined approach to orchestration. Pathway code looks similar to Python code for some of the above orchestrators. It also relies on a DAG which connects to data sources, merges data, transforms, and prints outputs to data sinks. Pathway runs on perpetually deployed machines which process live data streams and CDC data changes, with the ability to additionally schedule batch jobs.