Speed up Mammoth pipelines, refreshes and dashboards

Find what's slow

How dataset size affects speed

The more rows a task has to process, the longer it takes. On large datasets, reduce rows and columns early, and use Draft mode when you build or change several tasks at once.

Filter rows early in the pipeline

Mammoth runs your pipeline top to bottom, passing the full row set through each task in turn. Every row that exists at task 3 gets processed at task 4. So a Conditional Filter at task 2 that removes 70% of rows makes every later task process only 30% of the data.

Place Conditional Filter tasks as early as you can, ideally first or second after the source.

Slow:

Source: 2M rows
→ Join with customer table (2M × customer table)
→ Math Function: calculate margin (2M rows)
→ Conditional Filter: keep active accounts only (removes 60%)
→ Group & Pivot (800K rows)

Faster:

Source: 2M rows
→ Conditional Filter: keep active accounts only (→ 800K rows)
→ Join with customer table (800K × customer table, 60% less work)
→ Math Function: calculate margin (800K rows)
→ Group & Pivot (800K rows)

The filter moved from task 4 to task 1, so the Join works on 800K rows instead of 2M. Every later task gains too. In a second example, filtering 5M rows to the current year before a Join with a pricing table leaves 400K rows: the Join input drops by over 90%.

Combine several conditions into one Conditional Filter. One filter keeps the pipeline shorter and easier to read than several filters in a row.

Order pipeline tasks from light to heavy

Run the cheapest tasks first, so they shrink the data before the expensive tasks see it.

A good order for most pipelines:

1. Conditional Filter (reduce rows)
2. Remove Column or column selection (reduce width)
3. Data type conversions
4. Text standardisation (Bulk Replace, Text Formatting)
5. Enrichment (Join, Lookup)
6. Calculations (Math Function, Add Column)
7. Aggregation (Group & Pivot, Window function)

Remove unused columns right after the source

Wide datasets carry more data through every later task. Add a Remove Column task right after the source to keep only the columns your pipeline needs. For example, if a source has 47 columns and your analysis needs 10, removing the other 37 straight away means every later task handles less data.

Make joins faster and stop row fan-out

Joins are the most common bottleneck. Three rules apply:

1. Filter both datasets before joining. A Join compares each row in one dataset with the rows in the other that share the key. Reducing either input reduces the work.
2. Use Lookup instead of Join for single-column matching. Lookup fetches one value from another View by a matching key and returns at most one match per row, so it can't multiply rows. Use it, for example, to add a product name from a reference table. Use Join when you need several columns or specific Inner, Left, Right or Outer behaviour.
3. Check for duplicate join keys. If the key repeats on either side, the result multiplies rows (fan-out). Joining 100K rows to a reference table where each key appears 5 times gives 500K rows, and every later task processes 5 times the volume. Use Remove Duplicate Rows on the reference dataset before joining if needed.

Mammoth blocks a join that bloats the row count. A Join predicts its result size before it runs. When the predicted output is over 100,000 rows and more than 5 times the larger input's row count, the join fails with a row-count error instead of running. If you see that error, check both sides for duplicate keys, then remove duplicates or filter first. Lookup doesn't need this safeguard, which is another reason to prefer it for one-to-one enrichment.

Avoid joining to the same dataset repeatedly. Save the join result as a new dataset instead.

Apply several tasks at once with Draft mode

Draft mode (a pipeline setting that queues new tasks instead of running each one straight away) avoids re-running the full pipeline after every change. Use it when you build several tasks on a large dataset, redesign part of a pipeline, or test a new sequence of tasks.

1. Open the pipeline and click More Pipeline options (the ... menu in the pipeline toolbar).
2. Turn on Draft mode.
3. Add your tasks. They stay pending, and the pipeline shows Pending Updates.
4. Open Pending Updates. Click Review to see the changes, Apply now to apply them together, or Discard to drop them.

The pipeline runs once with all the new tasks applied. See How the pipeline works for more on Draft mode.

Split a long pipeline into stages

A pipeline with many tasks is faster to troubleshoot, and easier to maintain, when you split it across several Views. Each View handles one stage. Use an intermediate Send to Dataset task if several Views share a complex sub-pipeline.

Source dataset
└── View 1: Data quality (filter, clean, standardise)
    └── View 2: Enrichment (joins, lookups)
        └── View 3: Calculations and output

Each View can be monitored and run on its own. If a stage fails, the error stays in that stage.

Find the slow task in a pipeline

Use Preview Data at this Task on each task to see the row count at that point. Look for:

• tasks where the row count is higher than you expect (a filter that should cut rows but doesn't)
• tasks that work across the whole dataset, such as joins and Group & Pivot on large datasets
• Math Function columns with complex calculations across millions of rows

Then apply the matching fix:

Speed up a slow dashboard

A dashboard doesn't query its source Views each time someone opens it. New rows come into the dashboard's draft when it syncs (Sync new data: "Pull new rows from connected datasets into your draft"), and viewers see them after you publish. So a slow dashboard usually waits on one of these:

• the source View's pipeline run, before the dashboard can sync
• the sync itself, when a source View holds millions of unaggregated rows
• a publish: viewers see the last published version until you publish or turn on Publish new data automatically

Check the dashboard's sync status

1. Open the dashboard's Updates window, from the Dashboards library or the editor's state line.
2. Under Sync new data, check whether each source shows New data or Up to date, and when it last updated.
3. If you see "New source data is waiting to sync.", click Sync on each source marked New data. There is no single button for all sources.
4. Publish the dashboard.

See Manage your dashboards.

Aggregate and filter the View behind the dashboard

• Aggregate. For KPIs, totals and summary charts, the View should hold aggregated data, not row-level records. Add a Group & Pivot task at the end of the pipeline that summarises to the grain the dashboard needs. For monthly revenue by region, the View needs one row per month and region, not one per transaction.
• Pre-filter. Add a Conditional Filter task to limit the View to the time window or subset the dashboard shows. If the dashboard shows the last 12 months of sales, filter the View to the last 12 months, not all history.
• Speed up the source pipeline. If the source View's pipeline is the slow part, fix the pipeline with the steps above.

Speed up AI dashboard generation

Building a dashboard with AI takes a short while, because the AI reads your View's columns and data before it builds charts. See Create a dashboard. If the build seems stuck:

1. Refresh the page and try again.
2. If it keeps happening, simplify the prompt and reduce the number of columns in the source View. Narrow Views with a few focused columns generate faster than wide ones.

Pull less data from a Live Connection

When a Dataset Refresh runs, Mammoth queries your source and loads the result into the dataset. The whole pipeline on that dataset then re-runs on the new data. So two things control speed: how much data comes from the source, and how the pipeline processes it.

A pipeline's Conditional Filter runs on the copy of the data the refresh already pulled. It doesn't re-query your source. Filtering early still speeds up processing, but it doesn't reduce what your database sends on each refresh. Only two settings on the connection or automation do that: column selection and incremental refresh.

Select only the columns your pipeline uses

If a connection pulls all 45 columns from a wide table but your pipeline uses 8, you transfer 37 unused columns on every refresh. Request fewer columns at the connection, not in the pipeline.

• SQL connectors: edit the connection query to list only the columns your pipeline uses. Replace any SELECT *.
• App connectors (such as Salesforce): use field selection in the connector setup to limit the fields the API returns.

Pull only new or changed records with incremental refresh

For data that grows over time, such as transaction logs, event streams and CRM records, a full pull on every run wastes time. Incremental refresh pulls only new or changed records.

Native incremental pull (where your connector supports it): set the Update method to Add new data and choose a Key column, a column with increasing values such as an ID or a date. New data is added based on that column instead of replacing the whole dataset. Check your connector's setup screen for this option.

Pipeline fallback (when there's no native option): you need a Live Connection and permission to edit its dataset's pipeline.

1. Create a Dataset Refresh automation for the connection.
2. In the dataset's pipeline, add a Conditional Filter as the first task with a date condition, for example created_date > last_30_days, or a Mammoth parameter that tracks the last run time.
3. Schedule the automation as often as you need.

Each refresh then passes only recent rows through the rest of the pipeline. This fallback still pulls the full table from the source every time; it only reduces what flows through the pipeline. Prefer the native option whenever your connector has it. See Refresh a dataset.

Schedule refreshes so they don't compete

Refreshes that start at the same moment compete for the same source connections and processing capacity, so each runs slower. Large sources (ERP systems, databases, Salesforce) also have maintenance windows, rate limits and busy periods.

• Stagger start times for large datasets instead of running them all at midnight.
• Run related refreshes in sequence. When one dataset feeds another (for example, a customer master that feeds a transaction analysis), select both in one Dataset Refresh automation and choose In a sequence. They run one after another in the numbered order, which you can change by dragging. In parallel runs all of them at once.
• Know your plan's limit. How often an automation can run depends on your plan, and the schedule editor lists the units you can choose. A Workspace Owner can change the plan for more frequent refreshes. See Schedules and triggers.

To change an automation's schedule:

1. Open the Automations tab in the Data Library.
2. Click the automation, then click Edit in its detail panel.
3. Change the schedule.
4. Save the automation.

See Manage automations.

Fix a slow or failing refresh

If an automation fails, open it, read Last run result and fix the root cause. Then click Run now to re-run it. See Fix connection and refresh failures.

Speed up exports to BigQuery and SFTP

Export speed depends on the volume you export (millions of rows take longer), the network between Mammoth and the destination, and how fast the destination can write (some warehouse setups have write limits).

• Filter the dataset before exporting, so you send only the rows the destination needs.
• For growing data, add a date filter to export only rows that are new or changed since the last export, instead of the full table.
• For SFTP, check that the server isn't the bottleneck: look at its capacity and free disk space.

See Send to destinations.

Watch running work in Monitor

Open Monitor in the left sidebar. You don't need to stay on the page, because Mammoth runs processing on its servers.

See Monitor running, failing and paused pipelines and Activity log.

Limits that affect performance

Storage is set by your plan: see Workspace Settings → Plan & Storage for your allowance and per-dataset usage. Uploads from your computer are limited to 50 MB per request, and BI workbook imports (Tableau .twbx, Power BI .pbix) to 1 GB. A BigQuery query times out after 5 minutes, and a long-running job after 1 hour; contact Mammoth support if a connector's timeout interrupts your work. Contact your account team for the limits that apply to your workspace and plan, and see Limits and quotas.

Still slow? Contact support

1. Check Monitor: the Running tab shows whether a task is Stuck, and Needs Attention lists failed runs and errors.
2. Look for the slowest task. Join operations are the most common culprit.
3. Review Join steps for duplicate keys, because fan-out multiplies rows.
4. Move any Conditional Filter tasks earlier in the sequence.
5. Contact support@mammoth.io. Support can inspect server-side run logs. Include:
   • your workspace name and dataset name
   • the View's name and the slow task (or the pipeline ID)
   • the approximate row count and column count
   • the operation that's slow (dashboard load, pipeline run, export, automation)
   • how long it takes, and how long you'd expect

Related

• How the pipeline works: how tasks run, and Draft mode in detail.
• Join two datasets on a shared column: set up Join and Lookup.
• Refresh a dataset: create and schedule Dataset Refresh automations.
• Fix connection and refresh failures: when a refresh fails instead of running slowly.
• Limits and quotas: every size, row and plan limit.