Your Data Arrives Wrong. Here Is the Detection Layer.

Bad data does not announce itself.

It does not arrive with an error. It arrives looking exactly like good data — same columns, same row count, plausible numbers — and it sits in a report for three weeks until somebody in a meeting says a figure looks off, and the next hour is spent working out whether it is.

The problem is not that mistakes happen. It is that nothing is watching between the moment data arrives and the moment somebody notices.

There are two useful places to look. Once, deliberately, when you are holding a new dataset and want to know what you have got. And continuously, automatically, on data that arrives when nobody is looking at all. Those are different jobs and they need different tools.

Looking once: the quality report

When you have a dataset in front of you and want to know whether to trust it, the Data Quality Report runs an assessment against the DAMA framework — the Data Management Association's industry standard — and scores all six of its dimensions. It takes a minute or two.

The four that catch the most in practice:

Completeness measures the percentage of non-null values per column. It is the least interesting dimension and the one that most often explains a wrong total: a column that is 94% populated will happily produce an average that looks entirely reasonable.

Uniqueness finds duplicates and identifies key candidates — the columns actually suitable for joining. Duplicate records inflate metrics quietly, and a column you assumed was a key usually turns out not to be at the worst possible moment.

Timeliness compares the current batch's timestamps to the historical pattern. This is the one that catches yesterday's sales data still not having loaded at noon today — a failure with no bad values in it at all, which is exactly why nothing else catches it.

Validity checks type consistency and format. A date column where 3% of the values are text, a number column carrying a stray unit — the things that survive a glance and break an aggregation.

Report first, then decide. The point of a scored assessment is that you can look at a dataset you did not build and form a view in two minutes rather than an afternoon.

What the report cannot do

It cannot express your rules.

The six DAMA dimensions are the frame, and you cannot add a seventh. "An order line must have a delivery date within 90 days of the order date" is a real quality rule at a real company, and it is not a DAMA dimension — so the report will not check it, and no amount of configuration will make it.

That is a genuine boundary, not a gap we forgot to mention. The report tells you whether the data is structurally sound. It does not know your business.

Watching continuously: data checks

The report is something you run. A Data Check is something that runs whether or not you are there.

It is a task in the Transform → Controls category, placed at a specific point in a pipeline, and it evaluates a condition at runtime and produces a pass or fail. Unlike a Checkpoint — which is a human-in-the-loop approval gate — a Data Check is fully automated. Nobody has to be awake.

There are five built-in types, and they cover the DAMA dimensions without anyone writing SQL:

• Null Percentage — column, condition, threshold
• Row Count — at least, at most, between, exactly, or changed by more than, compared against the previous run
• Unique Values — 100% unique, or a percentage threshold
• Value Range — between, at least, at most, positive, non-negative, with a tolerance
• Custom Expression — where you describe the rule in plain language and it becomes DuckDB SQL you can read and edit directly

Row Count's "changed by more than" is the one to reach for first. Most broken feeds are not empty and not malformed. They are 40% smaller than last week because an upstream filter changed, and every other check passes. A check on the delta catches that. A check on the value never will.

And Custom Expression is where your business rules live. The delivery-date rule the quality report cannot express fits here, described in a sentence and turned into SQL you can inspect. If that same rule is about to exist in several pipelines, write it once rather than in each of them.

What happens when a check fails

This is the decision that matters, and it is yours to make per check.

Halt the pipeline. Downstream tasks stop and show as blocked — the same behaviour a refresh takes when the source changes shape. Nothing moves. Use this when the data is load-bearing — when a wrong number reaching a dashboard is worse than a late one.

Flag and continue. The check records a failure and shows a Failed pill, but the pipeline completes and downstream tasks run normally. Use this when you want to know without stopping the world — a soft signal on a feed where a small anomaly is common and blocking would cost more than it saves.

Most teams reach for halting on everything, and then turn it off after the third false alarm at 2am. It is worth deciding deliberately which of your checks are actually load-bearing, because a check everyone has learned to ignore is worse than no check at all.

The failures nobody catches by eye

Two arrive already wrong, before any check runs on them.

A decimal comma in a European export is read as a thousands separator unless something detects the locale. 1,234 becomes one thousand two hundred and thirty-four instead of one point two three four — a thousandfold error, on a structurally perfect file, with no null and no type violation for a check to catch.

Date formats do the same thing more quietly. 03/04/2026 is either March or April depending on which side of the Atlantic wrote it, and both readings are valid dates. Nothing is malformed. The rows simply land in the wrong month.

Detection on import exists because these two cannot be caught downstream. By the time the value is in a column, it is a plausible number and a valid date, and every check you could write on it passes.

Where to start

If you have a dataset you did not build and do not yet trust, run the quality report and read the four dimensions above. Two minutes.

If you have a feed that arrives on a schedule and matters, put one Data Check on it — a Row Count changed by more than check, set at whatever swing would genuinely surprise you. It is the single check that catches the most, and it is the one that would have caught the last quiet failure you had.

Everything else is refinement.