Pipeline error causes 40% wrong ad counts for 11 days

Eleven days. That was how long our pipeline ran perfectly with zero errors, green Directed Acyclic Graphs (DAGs), clean Snowflake loads while producing audience counts that were 40 % wrong. The system looked healthy, the dashboards stayed green, and no alerts popped up. Yet the numbers fed to advertisers were off by almost half.

The problem began when an upstream ad network quietly renamed a field in its event payload. A data pipeline is like a translation chain: if a word changes meaning at the source and nobody updates the dictionary, everything that follows is a confident mistranslation. In our case the Spark join key stopped matching. Segment counts collapsed, but the monitoring tools only checked for errors, not for correctness. A client noticed that their campaign was not delivering expected results and traced the issue back to that single field rename.

I was a Senior Data Engineer at InMarket, formerly NinthDecimal, when the glitch hit. We processed terabytes of advertising event data every day, handling over 1 PB of location and ad data. The system moved from MapR to S3, but the monitoring stayed the same. We trusted the green checkmarks and didn’t validate the numbers against business expectations. The incident exposed a blind spot in our quality checks.

Now the team is adding data‑quality alerts that compare output against historical baselines and business rules. They’re also tightening schema versioning so any upstream change triggers a pipeline rebuild. The lesson is clear: a pipeline that runs without errors can still produce wrong results, and the only way to catch that is by watching the numbers themselves.