Modern data workflow orchestration, part 1
Do we really need full blown orchestration services like Composer (managed Airflow) in a modern GCP data stack? Bundling vs Unbundling Airflow There’s been a lot of discussion about Airflow’s role in the Modern Data Stack lately and how different tools and services are unbundling the responsibilities of Airflow. It is a very interesting discussion, but to be fair to Airflow it was built as a workflow manager but its flexibility has invited users to add additional responsibilities resulting in a anti-pattern to fill the gap of a missing control plane across data tools/services....