Somebody asks where a dashboard number came from. And the answer is always the same: a pipeline carried it there.
I learned this the hard way. Early in my B2B days, I moved data between systems by hand: export, fix, import, repeat. Every Monday, for two hours. Then one week I forgot, and the sales report ran on week-old numbers. Nobody noticed for days. That scared me more than any error message.
So let’s talk about pipelines, the machines that do this job without forgetting.
📌 TL;DR: A data pipeline moves data automatically: extract from a source → transform into shape → deliver to a destination. Batch pipelines run on schedules. Streaming pipelines run non-stop. Build them once, monitor them forever.
What Are Data Pipelines?
A data pipeline is an automated series of steps that moves data from a source to a destination, usually transforming it on the way. No human hands, and no Monday exports. After that, the data just flows on its own.
Think of it like plumbing. You build the pipes once, and water arrives every time you open the tap. Same idea here, except the water is your CRM records, your events, your orders.
And here’s the thing. The pipeline itself is simple. Keeping it honest is the real job.
How does a pipeline actually work?
Every pipeline runs the same three moves. Let me walk you through them.
Extraction pulls data from the source. The smart question here: everything, or only what changed? Pulling everything is easy and wasteful. Grabbing only the changes is efficient. But you need a way to spot them. Timestamps and change-capture are the usual answers.
Transformation reshapes what arrived. Types get fixed and formats get standardized. Records get cleansed and joined. Most pipeline logic lives here. Most pipeline bugs live here too, so keep this code versioned and tested.
Loading writes the result to the destination. One rule saves you here: make it safe to run twice. Because someday it WILL run twice, and the destination should end up correct, not doubled.
→ Extract → transform → load → check. And that’s the whole machine, end to end.
What breaks in real pipelines?
Plenty. And the scary failures are the quiet ones. Here’s my short list:
- Schema drift: the source renames a column, and your transform breaks. Or worse, silently mismaps. Drift detection exists for exactly this.
- Silent partial loads: the run ‘succeeds’ with 40% of the data. Volume checks catch it. Nothing else will.
- Retry storms: a flaky source triggers retries that pile into the next run. Orchestration locks stop the pileup.
- Messy backfills: reprocessing old data with new logic makes patchwork history. Version your transforms.
- Cost creep: a pipeline built for a million rows meets a hundred million. The cloud bill tells you first.
Notice something? Not one of these throws a loud error. That’s why monitoring matters more than clever code.
A quick story about the quiet failure
Let me tell you about my favorite pipeline disaster. It wasn’t mine, thankfully. It belonged to a team I worked beside.
Their nightly load had been dropping 8% of rows for six weeks. No errors, no alerts, nothing. The source had added a new record type, and the transform skipped anything it didn’t recognize. Quietly and politely, every single night.
The dashboards looked fine. Just… slightly smaller. Revenue by region drifted down so gently that everyone blamed the market.
Then someone reconciled against the source. And the room went very quiet.
The fix took twenty minutes, but finding it took a day. But the six weeks of decisions made on shrunken data? Those never got un-made. So when I say volume checks matter more than clever code, that team is why. One row-count alert would have caught it on night one.
Set the alert before you need it. Seriously, it’s ten minutes of work.
Why do pipelines decide your data quality?
Because pipelines are where good data goes bad quietly. A perfect record delivered late is stale. Delivered twice, it’s a duplicate. And half-transformed, it’s a lie wearing a valid format.
So monitor three things on every flow: volume, freshness, and schema. Did roughly the right number of rows arrive? On time, and in the expected shape? Those three alerts catch most real-world failures.
And put your enrichment and cleansing steps INSIDE the pipeline. A pipeline that moves dirty data faster just spreads the mess more efficiently.
How mature is your pipeline practice?
Here’s a quick self-test. Be honest: each level includes the one before it:
- Cron scripts (level 1): failures found by users, knowledge in one person’s head
- Monitored jobs (level 2): alerts on failures and volumes, runs logged, a second operator possible
- Tested transforms (level 3): versioned code, output tests, backfills routine instead of adventures
- Orchestrated platform (level 4): dependency-aware scheduling, freshness promises per dataset, plus lineage you could show an auditor
Most teams sit at level 2 and believe they’re at 3. The tell? One simple backfill question. If reprocessing last month needs the person who wrote the pipeline, your tests aren’t real yet.
But don’t let that sting. Every level pays for itself fast, and none of them needs fancy tools. It just takes discipline, applied in the right order.
Real-World Examples
Pipelines feel abstract until you catch one in the wild. So here are three you’ve likely met without noticing.
The nightly sales feed. An online shop copies yesterday’s orders from the store database into a data warehouse at 2 a.m. Finance opens a fresh dashboard at 9. Nobody exported anything. That’s a classic batch processing pipeline.
The CRM sync. A form fill lands in a marketing tool. Ten minutes later the same contact appears in the CRM, cleaned up and matched to a company. A small scheduled pipeline did the ferrying.
The live product stream. Every click in an app flows into an events system, and an on-call dashboard updates within seconds. That’s the streaming flavor, running non-stop.
Different speeds, same three moves. And each one quietly decides what some team believes tomorrow.
Best Practices
You don’t need fancy tools to run pipelines well. But you do need habits. These are the ones that pay off first.
- Make every load rerun-safe. Runs repeat. Design the write so twice equals once.
- Alert on volume, freshness, and schema. Three checks, ten minutes each. They catch most silent failures, including shrinking row counts.
- Version your transforms. When logic changes, you want to know what touched last month’s numbers, and when.
- Keep runs small and boring. One job, one purpose. Clever multi-step monsters fail in clever multi-step ways.
- Write down the source owner. Because when a feed dies at 2 a.m., “who do we call?” shouldn’t be a research project.
None of this is glamorous. But boring pipelines are the ones still working a year from now.
How do pipelines relate to the neighbors?
A few cousins worth knowing. For scale, there’s the big data pipeline. Meanwhile the transform-order debate lives in ETL vs ELT. And when many pipelines need coordinating, that’s orchestration, the conductor for the whole orchestra.
Frequently Asked Questions
What is a data pipeline in simple terms?
A data pipeline is an automated route that moves data from where it’s created to where it’s used, cleaning and reshaping it on the way. Picture a conveyor belt between systems. It runs without human hands, and that’s the whole point.
What are the main stages of a data pipeline?
Three: extraction (pulling data from sources), transformation (cleaning and reshaping), and loading (writing to the destination). Monitoring wraps all three. Because the silent failures are the expensive ones.
What is the difference between a data pipeline and ETL?
ETL is one specific pipeline pattern: extract, transform, load. ‘Data pipeline’ is the umbrella over ETL, ELT, streaming flows, and plain copies. So every ETL job is a pipeline. But not every pipeline is ETL.
Do small teams need real pipelines?
Yes, sooner than they think. The moment two systems and one report depend on data moving daily, you need one. A small, boring, monitored pipeline costs an afternoon. The silent failure it prevents can cost a quarter.