Data Pipelines: Airflow, Prefect, and Orchestrating ETL at Scale
Orchestration is about dependencies, retries, and knowing what ran. Read the mechanism, an SLA worked example, and why event semantics matter more than scheduling.
A data pipeline is rarely difficult because of any single transformation. It becomes difficult when dozens of transformations depend on one another, run on different schedules, and fail in partial ways that leave downstream tables quietly wrong. Orchestration is the discipline of making those dependencies explicit and the failures visible.
Direct answer: Workflow orchestration tools — Apache Airflow and Prefect are the two named here — manage task dependencies, scheduling, retries, and observability for data workflows. CloudEvents provides "a specification for describing event data in a common way", which matters when pipelines react to events rather than clocks. GitHub Actions automates "software development workflows" and can run CI/CD, which overlaps with orchestration but is optimised for code, not data dependencies. The boundary: an orchestrator does not make a pipeline correct; it makes the order and state of execution explicit.

What an orchestrator actually provides
Four capabilities distinguish orchestration from a collection of cron jobs.
A dependency graph. Tasks are declared as a directed acyclic graph: this transformation runs after those two complete. The graph is the system's memory of how data is produced, which is valuable precisely when the original author has left.
Retry and failure semantics. A transient failure should be retried; a deterministic one should not. A good orchestrator lets you declare retries, backoff, and the conditions under which a task is considered failed.
State and history. You should be able to answer "did yesterday's load succeed?" and "which run produced this table?" without reading logs. That history is the basis of trust in a data warehouse.
Backfills. When a transformation changes, you need to recompute past periods in the correct order. A tool that cannot backfill reliably forces manual work and creates silent inconsistencies.
Scheduling versus events
Two trigger models appear in practice.
Scheduled pipelines run at fixed times — hourly, nightly. They are predictable and easy to reason about, and they assume the inputs are ready when the clock fires. When they are not, the pipeline either fails or processes stale data.
Event-driven pipelines run when something happens: a file arrives, an order is placed. CloudEvents exists because "the lack of a common way of describing events means developers have to write new event handling logic for each event source"; a standard format gives "accessibility" through common libraries and tooling and "portability" across environments. AMQP provides the messaging substrate for reliable delivery.
The two models are often combined. A nightly aggregate can be scheduled while a fraud feature is computed on each order event. The orchestrator's job is to make both models visible in one place, not to force a single trigger style.
A worked example of an SLA
Suppose a daily revenue table must be ready by 07:00, and three jobs feed it:
| Job | Duration | Depends on |
|---|---|---|
| Extract orders | 20 min | — |
| Extract refunds | 10 min | — |
| Join and aggregate | 15 min | both extracts |
Run sequentially, the total is 45 minutes, so a 06:15 start meets the 07:00 SLA with 15 minutes of slack. Run the two extracts in parallel, the critical path is 20 + 15 = 35 minutes, giving 25 minutes of slack.
Now add a retry. If the orders extract has a 10% chance of failing once and taking 20 minutes again, the expected duration is 20 + 0.1 × 20 = 22 minutes, and the 10% tail is 42 minutes on the critical path. With a 06:15 start, a single retry still finishes around 06:57 — just inside the SLA. Two retries would breach it.
This arithmetic explains three design choices:
- Parallelise independent extracts to buy slack before you need it.
- Set retry limits from the SLA, not from a default of three.
- Alert on predicted breach, not only on failure — a delayed input at 06:30 is already a problem even if nothing has failed yet.
The numbers are hypothetical, but the method is the point: an SLA is a budget of time, and retries, dependencies, and delays spend it.
Where CI/CD overlaps and where it does not
GitHub Actions is designed to "automate, customize, and execute your software development workflows", including CI/CD, with workflows, runners, and secrets. It can run a data transformation on a schedule, which is why teams sometimes use it as a lightweight orchestrator.
The difference is in the object being managed. CI/CD cares about code versions, test results, and deployments. Data orchestration cares about data partitions, freshness, upstream availability, and backfills. A tool optimised for the first can run a SQL job, but it will not tell you that a partition is stale or help you recompute six months of history in dependency order.
Use both, with clear boundaries: CI/CD to build and deploy the transformation code, the orchestrator to run it against data.
Idempotency is the safety property
A pipeline will be retried. A task will run twice after a worker loses its connection at the moment it was writing. The question is whether a second execution produces the same result as the first — the property called idempotency.
Three patterns make it achievable:
Overwrite by key. Instead of appending rows, write a partition keyed by the date being processed. A rerun replaces the partition rather than duplicating it. This is the most reliable pattern for batch aggregates.
Transactional writes. If the destination supports transactions, wrap the write so that a partial failure leaves no trace. The orchestrator's retry then starts from a clean state.
Deduplication keys. For event streams, attach a stable identifier to each event and make the destination reject duplicates. This is where CloudEvents' common format helps: the identifier travels with the event in a standard field rather than being invented by each consumer.
The alternative — hoping that retries are rare — fails at scale. A one-in-a-thousand duplicate rate on a million-row load produces a thousand wrong rows, and the error surfaces weeks later as a discrepancy nobody can reproduce.
What orchestration does not solve
Three limits matter.
A green task is not a correct table. An orchestrator confirms that a job exited successfully. It does not know whether the row count is plausible, whether a join silently dropped records, or whether a source changed its schema. Data-quality checks are a separate concern and should be part of the pipeline.
Dependencies must be declared. The orchestrator cannot infer that table B needs table A if nobody encoded it.
The graph rots. Pipelines accumulate tasks that nobody remembers the purpose of. Keep a Changelog's discipline — "all notable changes to this project will be documented in this file", with categories such as Added, Changed, and Removed — applies to pipeline definitions: record why a task was added and when it was retired.
A decision test
- Draw the dependency graph and identify the critical path.
- Parallelise independent work to buy SLA slack.
- Set retries and timeouts from the SLA, not from tool defaults.
- Add data-quality checks that run after each task, not just success/failure states.
- Use standard event formats such as CloudEvents when triggers are event-driven.
- Keep CI/CD and data orchestration separate with a clear handoff.
- Log changes to the pipeline with reasons and retirement dates.
For the operational side, see MLOps; for the broader architecture, see Microservices Architecture.
A counterargument
The article draws a firm boundary between an orchestrator and a CI/CD tool, then claims the orchestrator makes scheduled and event-driven work "visible in one place." A reasonable critic would say that boundary is a convention rather than a property established by the sources. CloudEvents and AMQP, as cited, specify an event envelope and a messaging protocol; neither supplies dependency graphs, run history, or backfills, so the "one place" is something the orchestrator adds rather than something the standards guarantee. GitHub Actions is described as automating "software development workflows," but the cited documentation does not say it cannot run a scheduled, retried data job - the claim that it "will not tell you that a partition is stale" is a feature gap, not an inherent limit.
The distinction is useful, yet the article presents a tool boundary as a principle.
References & Further Reading
- CloudEvents, 2026. CloudEvents — The vendor-neutral event envelope standard.
- OASIS, 2012. Advanced Message Queuing Protocol (AMQP) v1.0 - OASIS Open — The message-protocol standard behind most enterprise brokers.
- GitHub, 2026. GitHub Actions Documentation — The CI platform's own documentation, including required status checks.
- Olivier X, 2026. Keep a Changelog — The changelog format used by the project.