Find the step behind wrong pipeline output
When every task runs but the result is wrong, step through the pipeline with Preview Data at this Task until the output stops matching what you expect. The cause is usually task order, an expression, or a column stored as the wrong type. For tasks that show a red error, see Fix a failing step.
Step through the pipeline to find the wrong step
- Step through the pipeline. Use Preview Data at this Task on each task and compare it with what you expect.
- Check the task order. Tasks run in sequence. A Conditional Filter after a Join filters the joined output, not each dataset before the join. Reorder if needed (Edit, reorder and copy pipeline tasks).
- Check your expressions. Work out the result by hand for a few rows and compare.
Fix Group & Pivot and Conditional Filter surprises
| Symptom | Cause | Fix |
|---|---|---|
| Sum or Average won't work on a column | It holds numbers as text | Add Convert Column Type before Group & Pivot |
| Group & Pivot shows an empty group | The grouping column has empty values | Filter those rows out first if they don't matter, or handle the group later |
| Conditional Filter keeps rows you expected to remove | The conditions need all to be true, or only one | Check the And/Or logic |
| A filter on numbers behaves oddly | The numbers are stored as text | Convert the column type first |
status = "active" also drops empty rows | A condition excludes empty values too | Add a condition for empty values if you want to keep them |
Check an AI task that misses your intent
If an AI task (an Ask in plain words task, a Generative AI task, or a rule written in a function's AI bar) runs but misses your intent:
- Read its prompt and check the output on a few rows.
- Make the prompt more specific. Start simple and add detail step by step.
- Compare it with a manual task and keep whichever matches your logic.
Note prompts that work so you can reuse them.
Prevent pipeline errors
- Filter early to reduce size and complexity (Speed up pipelines and connectors).
- Check data types before calculations, and handle empty values on purpose. Don't assume there are none.
- Use conditions to route problem rows into a separate flow.
- Test each task on small, known data with edge cases (empty values, duplicates, extremes), and check it with Preview Data at this Task.
- Test the whole pipeline on realistic volumes and source variations. Use Draft mode to queue tasks and review them before they take effect (How the pipeline works).
- Add a Data Check between major stages as a quality gate. It can halt the pipeline and block later tasks, or warn and continue, and notify the people you choose. When a check fails, its result shows how many checks failed: fix the data and run again, or change the check threshold. See Remove duplicate rows and add data quality checks.
- Compare row counts at key stages, and review the Data Quality report after changes. See Validation for patterns.
Related
- Fix a failing step: tasks with a red error.
- Fix a join that returns zero or too many rows: key problems.
- Data quality report: find nulls, duplicates and type problems before they break a step.
Was this page helpful?
Something wrong? support@mammoth.io