> AI agents: this is one page from Mammoth Analytics documentation. The index of all pages as Markdown is https://mammoth.io/docs/llms.txt. Append `.md` to any docs URL, or send `Accept: text/markdown`, to get Markdown.

# Speed up slow Mammoth data refreshes and exports

> Make Mammoth refreshes and exports faster: pull fewer columns, use incremental refresh, stagger schedules, fix timeouts and speed up BigQuery and SFTP.
>
> Applies to: Where: Data Library › Automations › Edit

When a Dataset Refresh runs, Mammoth queries your source, loads the result into the dataset, then re-runs the dataset's whole pipeline. Speed depends on how much data comes from the source and how the pipeline processes it. A pipeline's **Conditional Filter** runs on the copy the refresh already pulled: it speeds up processing but doesn't reduce what your database sends. Only two connection or automation settings do that: column selection and incremental refresh.

## Select only the columns your pipeline uses

If a connection pulls all 45 columns from a wide table but your pipeline uses 8, you transfer 37 unused columns on every refresh. Request fewer at the connection:

- **SQL connectors:** edit the connection query to list only the columns you use. Replace any `SELECT *`.
- **App connectors (such as Salesforce):** use field selection in the connector setup to limit the fields the API returns.

## Pull only new or changed records with incremental refresh

For data that grows over time (transaction logs, event streams, CRM records), incremental refresh pulls only new or changed records.

**Native incremental pull (where your connector supports it):** set the **Update method** to **Add new data** and choose a **Key column** with increasing values, such as an ID or a date. New data is added based on that column instead of replacing the whole dataset. Check your connector's setup screen for this option.

**Pipeline fallback (when there's no native option):** you need a Live Connection and permission to edit its dataset's pipeline.

1. Create a Dataset Refresh automation for the connection.
2. Add a **Conditional Filter** as the dataset pipeline's first task with a date condition, for example `created_date > last_30_days`, or a Mammoth parameter that tracks the last run time.
3. Schedule the automation as often as you need.

The fallback still pulls the full table from the source every time; it only reduces what flows through the pipeline. Prefer the native option. See [Refresh a dataset](https://mammoth.io/docs/learn/automate/refresh-a-dataset/).

## Schedule refreshes so they don't compete

Refreshes that start at the same moment compete for source connections and processing capacity. Large sources (ERP systems, databases, Salesforce) also have maintenance windows, rate limits and busy periods.

| Time window | Recommended use |
| --- | --- |
| Business hours | Light refreshes only; critical updates |
| Evening or overnight | Large refreshes, historical backfills, full-table pulls |
| Weekends | Weekly aggregations, full refreshes |

- **Stagger start times** for large datasets instead of running them all at midnight.
- **Run related refreshes in sequence.** When one dataset feeds another (for example, a customer master that feeds a transaction analysis), select both in one Dataset Refresh automation and choose **In a sequence**. They run in the numbered order, which you can change by dragging. **In parallel** runs all of them at once.
- **Know your plan's limit.** How often an automation can run depends on your plan, and the schedule editor lists the units you can choose. A Workspace Owner can change the plan for more frequent refreshes. See [Schedules and triggers](https://mammoth.io/docs/learn/automate/schedules-and-triggers/).

To change a schedule, open the **Automations** tab in the Data Library, click the automation, click **Edit** in its detail panel, change the schedule and save. See [Manage automations](https://mammoth.io/docs/learn/automate/manage/).

## Fix a slow or failing refresh

| Symptom | Likely cause | Fix |
| --- | --- | --- |
| Dataset Refresh fails with a timeout error | The source query pulls too much data, or the source is under load. Timeouts vary by connector: BigQuery allows up to 5 minutes for a query and up to 1 hour for a long-running job | Switch to incremental refresh, request fewer columns, and index the source columns your query or incremental pull uses (unindexed columns cause full table scans). If timeouts persist, contact Mammoth support with the connection ID |
| A refresh that took 3 minutes now takes 15 | The source table grows and you pull the full table each time | Switch to incremental refresh (native, or the pipeline fallback) |
| An automation takes much longer than before | The source dataset grew a lot, a recently added task processes the full dataset slowly, or other automations run at the same time | Check the dataset's size and recent pipeline edits, and stagger the schedules |
| The same automation fails now and then | Source rate limits, maintenance windows or brief network problems | Reschedule around maintenance windows and busy periods. Check **Monitor** for failed runs and re-run once the source is available |

If an automation fails, open it, read **Last run result**, fix the root cause, then click **Run now**. See [Fix connection and refresh failures](https://mammoth.io/docs/learn/help/connection-failures/).

## Speed up exports to BigQuery and SFTP

Export speed depends on the volume (millions of rows take longer), the network between Mammoth and the destination, and how fast the destination writes (some warehouse setups have write limits).

- Filter the dataset before exporting, so you send only the rows the destination needs.
- For growing data, add a date filter to export only rows new or changed since the last export.
- For SFTP, check the server's capacity and free disk space.

See [Send to destinations](https://mammoth.io/docs/learn/share-and-export/send-to-destinations/).

## Related

- [Speed up Mammoth pipelines](https://mammoth.io/docs/learn/help/performance/): the pipeline that re-runs after each refresh.
- [Fix connection and refresh failures](https://mammoth.io/docs/learn/help/connection-failures/): when a refresh fails instead of running slowly.
- [Limits and quotas](https://mammoth.io/docs/learn/help/limits/): every size, row and plan limit.

---
Source: https://mammoth.io/docs/learn/help/slow-refreshes.md · Updated: 2026-11-08