Performance
Mammoth runs all processing server-side, so your browser is not the bottleneck. How you structure your pipelines directly affects how fast they run.
This section is for analysts and technical users working with large datasets. This page covers the highest-impact optimizations and the platform limits. Start with Filter early, then go deeper in the child pages:
- Pipeline optimization: Order tasks, optimize joins, and use Draft mode.
- Connector performance and best practices: Reduce the data a Live Connection transfers and schedule refreshes.
Dataset size guidance
The more rows a task has to process, the longer it takes. On large datasets, reduce rows and columns early, and use Draft mode when you build or change several tasks at once.
| Dataset size | Recommended approach |
|---|---|
| Smaller datasets | Standard pipelines work well with no special optimization |
| Millions of rows | Apply Conditional Filter and Remove Column tasks early; aggregate for dashboards |
| Very large datasets | Use cloud connectors (such as BigQuery or Snowflake) rather than file upload; aggregate Views before building dashboards. Discuss architecture with support@mammoth.io if performance is still unacceptable |
The highest-impact optimization: Filter early
The earlier you reduce your row count in a pipeline, the faster every subsequent task runs.
Slow pattern:
All 5M rows → Join with pricing table → Conditional Filter: keep current year only
Optimized pattern:
All 5M rows → Conditional Filter: keep current year only → Join with pricing table (now 400K rows)
Moving the filter before the join reduces the join input by over 90% in this example, so the join runs proportionally faster.
Pipeline optimization checklist
| Optimization | When it helps |
|---|---|
| Filter rows early | Any pipeline with joins, pivots, or Math Functions on large datasets |
| Remove unused columns early | After importing wide datasets you don't need all of. Add a Remove Column task right after the source |
| Use Draft mode for multi-task changes | When building several new tasks on a large dataset |
| Use Lookup instead of Join for single-column matching | Lookup retrieves one value from another View by a matching key and returns at most one match per row, so it can't multiply rows. Use Join when you need several columns or specific Inner, Left, Right, or Outer behavior |
| Avoid repeated joins to the same dataset | Cache the join result as a new Dataset instead |
| Schedule refreshes off-peak and stagger them | For automations refreshing large Live Connection sources. Refreshes that start at the same moment compete for resources, so see Scheduling |
Draft mode
Draft mode lets you queue multiple pipeline tasks and run them all at once, rather than executing each task immediately as you add it. This avoids re-running the full pipeline after every individual change.
When to use it: Building or redesigning pipelines on large datasets, or when testing a sequence of new tasks before committing.
How to enable: Open the pipeline view and turn on Draft mode in the More Pipeline options menu (the ... menu in the pipeline toolbar). New tasks stay pending, and the pipeline shows Pending Updates. Open it and click Apply now to apply them together, or Discard to drop them.
Live Connections and incremental refresh
Live Connections (databases, APIs) query your source system at refresh time. Two optimizations matter most:
- Incremental refresh: pull only new or changed records instead of the full table on every refresh, either through a connector's native incremental-pull configuration or a parameter-driven Conditional Filter as a fallback
- Column selection: request only the columns your pipeline actually uses at the connection level, not the pipeline level
A pipeline-level Conditional Filter reduces processing time once data has arrived, but does not reduce how much data Mammoth requests from your source system. See Connector performance and best practices for the full breakdown.
Related: Dataset Refresh and Automations.
Monitoring with Monitor
When a pipeline or automation is running, Monitor (in the left sidebar) shows:
- Needs Attention: errors, pending data, and governance items (pipeline checkpoints, failed data checks) waiting on you
- Running: pipelines, tasks, and automations currently executing
- Paused: pipeline tasks, exports, and connections with Auto-sync off
- Activity Log: a chronological event history (visible if your role has activity log access)
You don't need to stay on the page. Mammoth runs processing server-side.
See Monitor for details.
Platform limits reference
| Resource | Guidance |
|---|---|
| Storage | Limited by your plan. See Workspace Settings → Plan & Storage for your allowance and per-dataset usage. |
| File upload size | Uploading files from your computer is limited to 50 MB per upload request. BI workbook imports (Tableau .twbx, Power BI .pbix) support up to 1 GB. See limits and quotas. |
| BigQuery query timeout | 5 minutes per query, and 1 hour for a long-running job. Contact Mammoth support if a connector's timeout is interrupting your workflow. |
Contact your account team for the limits that apply to your workspace and plan.
When performance issues persist
If a pipeline is consistently slow after applying these optimizations, work through Performance issues to diagnose the slow task and contact support.