Modern data workflow orchestration, part 2
The previous post gave some background on why you should try out GCP Workflows. This post is more technical and shows how to use Infrastructure as Code to easily set up GCP Workflows that also supports re-runs and backfills. Solution Architecture The requirements we have are: We want re-run capabilities (idempotency) and backfill capabilities. We will use a custom cloud run service built with FastAPI. We want to set up workflow orchestration and scheduling of batch jobs with Infrastructure as Code (IaC)....