Speed up slow Mammoth data refreshes and exports
When a Dataset Refresh runs, Mammoth queries your source, loads the result into the dataset, then re-runs the dataset's whole pipeline. Speed depends on how much data comes from the source and how the pipeline processes it. A pipeline's Conditional Filter runs on the copy the refresh already pulled: it speeds up processing but doesn't reduce what your database sends. Only two connection or automation settings do that: column selection and incremental refresh.
Select only the columns your pipeline uses
If a connection pulls all 45 columns from a wide table but your pipeline uses 8, you transfer 37 unused columns on every refresh. Request fewer at the connection:
- SQL connectors: edit the connection query to list only the columns you use. Replace any
SELECT *. - App connectors (such as Salesforce): use field selection in the connector setup to limit the fields the API returns.
Pull only new or changed records with incremental refresh
For data that grows over time (transaction logs, event streams, CRM records), incremental refresh pulls only new or changed records.
Native incremental pull (where your connector supports it): set the Update method to Add new data and choose a Key column with increasing values, such as an ID or a date. New data is added based on that column instead of replacing the whole dataset. Check your connector's setup screen for this option.
Pipeline fallback (when there's no native option): you need a Live Connection and permission to edit its dataset's pipeline.
- Create a Dataset Refresh automation for the connection.
- Add a Conditional Filter as the dataset pipeline's first task with a date condition, for example
created_date > last_30_days, or a Mammoth parameter that tracks the last run time. - Schedule the automation as often as you need.
The fallback still pulls the full table from the source every time; it only reduces what flows through the pipeline. Prefer the native option. See Refresh a dataset.
Schedule refreshes so they don't compete
Refreshes that start at the same moment compete for source connections and processing capacity. Large sources (ERP systems, databases, Salesforce) also have maintenance windows, rate limits and busy periods.
| Time window | Recommended use |
|---|---|
| Business hours | Light refreshes only; critical updates |
| Evening or overnight | Large refreshes, historical backfills, full-table pulls |
| Weekends | Weekly aggregations, full refreshes |
- Stagger start times for large datasets instead of running them all at midnight.
- Run related refreshes in sequence. When one dataset feeds another (for example, a customer master that feeds a transaction analysis), select both in one Dataset Refresh automation and choose In a sequence. They run in the numbered order, which you can change by dragging. In parallel runs all of them at once.
- Know your plan's limit. How often an automation can run depends on your plan, and the schedule editor lists the units you can choose. A Workspace Owner can change the plan for more frequent refreshes. See Schedules and triggers.
To change a schedule, open the Automations tab in the Data Library, click the automation, click Edit in its detail panel, change the schedule and save. See Manage automations.
Fix a slow or failing refresh
| Symptom | Likely cause | Fix |
|---|---|---|
| Dataset Refresh fails with a timeout error | The source query pulls too much data, or the source is under load. Timeouts vary by connector: BigQuery allows up to 5 minutes for a query and up to 1 hour for a long-running job | Switch to incremental refresh, request fewer columns, and index the source columns your query or incremental pull uses (unindexed columns cause full table scans). If timeouts persist, contact Mammoth support with the connection ID |
| A refresh that took 3 minutes now takes 15 | The source table grows and you pull the full table each time | Switch to incremental refresh (native, or the pipeline fallback) |
| An automation takes much longer than before | The source dataset grew a lot, a recently added task processes the full dataset slowly, or other automations run at the same time | Check the dataset's size and recent pipeline edits, and stagger the schedules |
| The same automation fails now and then | Source rate limits, maintenance windows or brief network problems | Reschedule around maintenance windows and busy periods. Check Monitor for failed runs and re-run once the source is available |
If an automation fails, open it, read Last run result, fix the root cause, then click Run now. See Fix connection and refresh failures.
Speed up exports to BigQuery and SFTP
Export speed depends on the volume (millions of rows take longer), the network between Mammoth and the destination, and how fast the destination writes (some warehouse setups have write limits).
- Filter the dataset before exporting, so you send only the rows the destination needs.
- For growing data, add a date filter to export only rows new or changed since the last export.
- For SFTP, check the server's capacity and free disk space.
See Send to destinations.
Related
- Speed up Mammoth pipelines: the pipeline that re-runs after each refresh.
- Fix connection and refresh failures: when a refresh fails instead of running slowly.
- Limits and quotas: every size, row and plan limit.
Something wrong? support@mammoth.io