If you run more than a hundred dbt models, you have probably had a morning like this: one source table arrives late, a staging model fails, and every model downstream of it fails too. Your alerting tool, doing exactly what it was told, sends a message for each one.
The alerts aren’t wrong. They are just shaped like your DAG instead of like the problem. Forty messages describe one cause, and the most useful one, the first failure, is usually buried in the middle.
Finding the first domino
The fix is to treat failures as a graph problem. After each run we take every failed or skipped node and walk upstream until we reach a node that either succeeded or failed for its own reasons. That node is the candidate cause. Everything downstream of it that failed in the same run joins the same incident.
- A node that failed with its own error, and whose parents all succeeded, is a cause.
- A node that was skipped or failed because a parent failed is a symptom.
- Two causes in one run become two incidents, even if their blast radius overlaps.
The hard cases
It gets interesting when nothing fails. A source that arrives with half its usual rows won’t break anything, but every model downstream will be quietly wrong. That’s why we treat volume and freshness anomalies as possible causes too, not just errors.
The useful question isn’t what failed. It’s what is the earliest thing that changed.
What it does to your channel
Across customers in their first month, grouping by cause cuts the alerts per incident by 68%. More importantly, the one alert that remains names the model to look at, so the first ten minutes of every incident go to fixing instead of reading logs.
If you want to see what your own alert history looks like grouped by cause, we can run it during a demo on a copy of your project.
