Fix duplicates, missing values and outliers
The Data Quality Report never changes your data or adds tasks: you build the fixes as steps in the pipeline. Each fix below matches a low dimension score. See How the pipeline works for adding steps, and Data quality report for finding the problems.
Remove duplicate rows (low Uniqueness)
Duplicates usually come from a source exporting the same records twice, a refresh running twice before deduplication, or files combined without deduplication.
- Add a Remove Duplicate Rows step after your data source.
- Under Evaluate duplicates from, choose the columns that decide whether two rows match (for example transaction ID, or customer ID + date).
- Apply the step, then compare the row count before and after.
More: Remove duplicates and check data.
Handle missing values (low Completeness)
Nulls come from optional fields left empty, left joins with no match, or cells skipped on import. Before removing null rows, check what share of the data they are: a high null rate often means a collection problem at the source.
| Strategy | When to use | Pipeline step |
|---|---|---|
| Replace nulls with a default | 0 is meaningful, or a text placeholder is acceptable | Fill Missing Values: choose the column and how to fill it |
| Remove rows with nulls | Analysis needs complete records | Conditional Filter: keep rows where each critical column has a value |
| Leave nulls | Null means "not applicable" | None; note what null means in the dataset description |
Fix data type mismatches
Dates stored as text ("January 15, 2024"), numbers with symbols ("$1,250.00"), or "Yes"/"No" in place of true/false make sorting, filtering and calculations misbehave.
- Add a Convert Column Type step.
- Select the column and choose the target type.
- For dates, give the format the text is written in.
- For numbers with currency symbols, first add a Find & Replace step to strip them, then convert.
More: Column types.
Deal with outliers (Accuracy)
Outliers can be real extremes or errors, such as 99999 entered as a placeholder. Check the actual values before removing anything, and ask the data owner: removing valid outliers makes analysis misleading.
| Strategy | When to use | Pipeline step |
|---|---|---|
| Remove outlier rows | Entry errors or test data | Conditional Filter: keep rows below an upper bound you choose after investigating |
| Cap outlier values | Real values that should be limited for analysis | Math Function in SQL mode, for example CASE WHEN amount > 10000 THEN 10000 ELSE amount END |
| Flag and keep | Real, meaningful values to mark | Label & Insert Values to label rows meeting a condition, or keep them and filter in the dashboard |
Standardize inconsistent text (Consistency)
- Add a Text Formatting step and use Change text case to normalize case.
- Add a Find & Replace step to map variants such as "USA" and "United States" to "US" (repeat for each variant).
- Add a Text Formatting step with Normalize whitespace to remove extra spaces.
More: Work with text and JSON and Find and replace values.
Apply several fixes in the right order
- Types first: null counts and outlier detection depend on the right type.
- Duplicates second: so filling nulls doesn't create false completeness.
- Nulls third: decide which to fill or remove.
- Standardize values last.
Generate the report again after each group of fixes, or after every 3–4 tasks, to track progress and avoid over-fixing.
Related
- How the data quality score is calculated: which dimension each fix raises.
- Filter rows: the Conditional Filter step for nulls and outliers.
- Fix a failing step: when a fix breaks the pipeline.
Something wrong? support@mammoth.io