Pipeline optimization

Two techniques account for most performance gains: filter early and sequence transforms from light to heavy. Everything else refines those two principles. This page is for analysts and technical users who build pipelines on large datasets. Start with Filter early.

Understanding pipeline execution

Mammoth executes your pipeline top to bottom, passing the full row set through each transformation in sequence. Every row that exists at task N gets processed at task N+1. Row count at each stage directly controls how much work later tasks must do.

The implication: A Conditional Filter at task 2 that removes 70% of rows makes every task after task 2 process only 30% of the data.

Core principle 1: Filter early

Place Conditional Filter tasks as early as possible in your pipeline—ideally as the first or second transformation after Source.

Slow:

Source: 2M rows
→ Join with customer table (2M × customer table)
→ Math Function: calculate margin (2M rows)
→ Conditional Filter: keep active accounts only (removes 60%)
→ Group & Aggregate (800K rows)

Optimized:

Source: 2M rows
→ Conditional Filter: keep active accounts only (→ 800K rows)
→ Join with customer table (800K × customer table — 60% less work)
→ Math Function: calculate margin (800K rows)
→ Group & Aggregate (800K rows)

The filter moved from task 4 to task 1. The Join now operates on 800K rows instead of 2M, a 60% reduction in join cost. Every later task benefits.

Combine multiple conditions into one filter. A single Conditional Filter with several conditions keeps the pipeline short and easier to read than several separate filters in a row.

Core principle 2: Sequence transforms from light to heavy

Order transformations so the cheapest operations run first and reduce the dataset before expensive operations process it.

Category	Transforms	Why
Light—run early	Conditional Filter, Remove Column, Convert Column Type	Reduce rows or columns, or change a type
Medium	Text Formatting, Label & Insert Values, Fill Missing Values, Bulk Replace	Work through every remaining row
Heavy—run late	Join, Lookup, Group & Pivot, Window function, Math Function on many columns	Combine tables or compute across rows

Recommended sequence for most pipelines:

1. Conditional Filter (reduce rows)
2. Remove Column / Column selection (reduce width)
3. Data type conversions
4. Text standardization (Bulk Replace, Text Formatting)
5. Enrichment (Join, Lookup)
6. Calculations (Math Function, Add Column)
7. Aggregation (Group & Pivot, Window function)

Core principle 3: Remove unused columns early

Wide datasets carry more data through every later task. Use Remove Column right after Source to keep only the columns your pipeline actually needs.

Example: A source with 47 columns but your analysis needs only 10. Removing the 37 unused columns immediately after Source means every downstream task handles less data.

Join optimization

Joins are the most common performance bottleneck. Three rules apply:

1. Filter both datasets before joining.
The Join operation processes every row in dataset A against every row in dataset B that shares the join key. Reducing either input directly reduces join work.

2. Use Lookup instead of Join for single-column key matching.
When you're matching on a single column and expect one-to-one matches (for example, adding a product name from a reference table), Lookup avoids accidental row multiplication.

3. Check for duplicate join keys.
If your join key has duplicates on either side, the result can multiply rows (a "fan-out"). A dataset with 100K rows joined to a reference table where each key appears 5 times produces 500K rows—every later task processes 5× the expected volume. Use Remove Duplicate Rows on the reference dataset before joining if needed.

Mammoth rejects a join that bloats the row count. Join estimates the resulting row count before running. When the projected output exceeds both 100,000 rows and 5× the larger of your two input row counts, the join fails with a row-count error instead of running. If you see that error, check for duplicate keys on both sides and remove duplicates or filter first. Lookup doesn't need this safeguard—it's designed to always return at most one match per input row, so it can never fan out row counts, which is the other reason to prefer it over Join for simple one-to-one enrichment.

Draft mode

Draft mode keeps new pipeline tasks pending and applies them all at once when you're ready, instead of running the full pipeline after every individual addition.

When Draft mode helps:

• Building several new tasks on a large dataset
• Redesigning a section of an existing pipeline
• Testing a new transformation sequence before committing

How it works:

1. Open the pipeline view and turn on Draft mode in the More Pipeline options (...) menu in the pipeline toolbar
2. Add your new transformations. They stay pending and don't take effect yet.
3. When ready, open Pending Updates and click Apply now. The pending tasks are applied together. Review shows the pending changes, and Discard drops them.

Expected result: The pipeline runs once with all the new tasks applied.

Structuring long pipelines

Pipelines with many tasks benefit from splitting into multiple views, where each view handles a distinct processing stage—both for performance and for maintainability. Use intermediate Send to Dataset tasks if a complex sub-pipeline is shared by multiple views.

Architecture pattern:

Source dataset
└── View 1: Data quality (filter, clean, standardize)
    └── View 2: Enrichment (joins, lookups)
        └── View 3: Calculations and output

Each view can be independently monitored and run. This also makes troubleshooting easier—if a stage fails, the error is isolated.

Monitoring pipeline performance

Open Monitor from the left sidebar. The Running tab shows pipelines, tasks, and automations that are executing or queued, and flags a task as Stuck if it runs unusually long. The Needs Attention tab lists failed runs.

When a pipeline is consistently slow

1. Check Monitor for a Stuck status or failed runs
2. Look for the slowest task: Join operations are the most common culprit
3. Check for duplicate join keys that might be causing row fan-out
4. Move any Conditional Filter tasks earlier in the sequence
5. Contact Mammoth support with the pipeline ID if the issue persists. The support team can inspect server-side execution logs

See Monitor for details.