Batches in Mammoth: see, suspend and delete what arrived
A batch is one load of data into a dataset: an uploaded file, a connector refresh or a consolidated file. Each batch keeps its name, date, row count and source, so you can see what arrived, take a batch out with Suspend, or remove it with Delete. Open the dataset's panel in the Data Library, then Settings & history, then Batches.
How batches make up a dataset
One dataset can hold many batches, each a separate source or time period. All batches share the same columns, Views show all batches combined by default, and pipeline steps apply to every batch.
Sales Dataset
├── April_Sales_North.csv (127 rows) - Batch 1
├── April_Sales_South.csv (143 rows) - Batch 2
└── June_Sales_Combined.csv (298 rows) - Batch 3
Total Dataset: 568 rows across 3 batches
Batches give you an audit trail (what came from where and when), versions you can roll back or compare without separate backups, and analysis by batch or source inside one dataset.
Where batches come from
Batches are created for you. Whether a new load adds a batch or replaces the old ones depends on the dataset's Update method (see Choose add or replace).
| Source | What it adds |
|---|---|
| File upload or Add data | One batch per file, named after the file, with the time captured |
| Data Consolidation automation | One batch per new file in a monitored Data Library folder (not an external folder), named from the file name, after a schema check |
| Dataset Refresh automation | Pulls the latest data from a connector on a schedule (daily, weekly or monthly). In Combine mode, a new batch per refresh, keeping history. In Replace mode, the previous batches are deleted and one new batch is added; only their details stay in the Activity Log. |
To set up Dataset Refresh: go to Automations, then Dataset Refresh; select the dataset(s) with a live connection; set the schedule (frequency, weekday, start and end date); and for several datasets choose Sequential or Parallel. Sequential lets you drag to set the order.
To set up Data Consolidation: go to Automations, then Data Consolidation; select a source folder to monitor; select the destination dataset (optionally filtered by file name pattern); then review and save. Combine or Replace is not chosen here: it is the destination dataset's Update method, which you can change at any time. When a file lands, Mammoth checks its columns, adds it as a new batch, runs the pipeline, and delivers exports to their destinations.
The Activity Log records every batch operation, manual or automated: additions, suspend and unsuspend, deletions and data retention runs. See Activity log.
Read the batch list
The list shows 10 batches per page; use the previous and next arrows to move between pages.
| Field | What it shows |
|---|---|
| Name | The source file name or a custom name. Select it to preview the batch. |
| Date | The exact creation date and time |
| Row Count | Rows this batch contributes |
| Source | An icon for where the batch came from: a file upload, a database connector, or a Combine Dataset operation |
| State | Active, Suspended, or Pending Delete (waiting for approval under a retention policy) |
| Action | What it does |
|---|---|
| Delete | Permanently deletes the selected batches. This can't be undone. |
| Suspend / Unsuspend | Takes batches out of use, or brings them back, without deleting them |
| Add data | A button below the list (file-based datasets only) that opens the Add data dialog |
A batch that reads Applying recipe or Pending is still being added: see Fix a pending batch.
Suspend a batch or delete it
Suspend removes a batch from Views and pipeline runs straight away, but keeps all its data and details. Select it again and choose Suspend / Unsuspend to bring it back. Use it to test the impact of a batch, quarantine bad data, leave out a season, or prepare a rollback.
Delete removes a batch permanently after you confirm ("Are you sure you want to delete the selected batches?"). Use it only when the data is no longer needed for analysis or compliance. A retention policy can also delete batches for you.
| Property | Suspend | Delete |
|---|---|---|
| Data kept | Yes | No |
| Details kept | Yes | Yes |
| Recovery | Instant, with one click | Not recoverable in the app. Batches removed by a retention policy are also permanent; their details stay in the Activity Log |
| Best use | Temporary exclusion, testing | Permanent exclusion |
Views that don't update after a new batch
Auto-sync is a setting on each View's incoming link. When it is off, the View is not rerun when its dataset gets a new batch, and the new data stays pending. A warning triangle (⚠) with "Data may be inconsistent" appears in the dataset's Views list, and Monitor's Needs Attention tab shows "new batch received, auto-sync off" with Preview and Process buttons. See Views: create, rename and delete.
Keep batches easy to work with
- Use Replace for latest-only data, so batches don't pile up.
- Set up a retention policy to delete or suspend old batches by age, count or condition.
- Name files before upload, because batch names come from file names.
Sales_North_Region_Q2_2024.csvis descriptive and sortable;data_export_final_v3.csvhas no context or date. A pattern such asLocation_YYYY_MM_DD.csvmakes batches easy to filter. - Apply a Conditional Filter early in pipelines on large datasets to reduce processing.
| Limit | Value |
|---|---|
| Batch size from a file upload | Up to 50 MB per file (see Limits and quotas) |
| Batches per page in the list | 10 |
Troubleshooting batches
| Symptom | Cause | Fix |
|---|---|---|
| Columns don't line up when combining ("Schema mismatch: column mapping required") | Column names differ (case and spacing count), types are incompatible, or columns are missing | Check names exactly, map columns in the pending batch's Column Mapping tab, and use the same export template for every source |
| Unexpected row counts | Combine or Replace wasn't what you expected, or batches duplicate each other | Check the dataset's update method and each batch's row count |
| A batch isn't in the analysis | It is suspended, or transformations didn't rerun | Check it shows Active, and that the pipeline reran after the batch was added |
| Historical data is missing after adding a batch | Replace was selected (can't be undone), batches were suspended, or a retention policy removed them | Check suspended batches and the retention policy, and review the Activity Log for removals |
| Data doesn't look updated after Replace | Your browser shows an old copy | Hard-refresh (Ctrl+Shift+R on Windows, Cmd+Shift+R on Mac) and check the Activity Log shows the Replace completed |
| A suspended batch still shows in a View | The status hasn't refreshed | Check the batch shows as suspended, close and reopen the dataset, and clear cached pipeline results. If it still appears, contact support. |
| Data Consolidation doesn't combine files | The file landed in the wrong place, failed the schema check, or the automation didn't run. A file with different columns is added as a separate file. | Check the Activity Log for the run, confirm the folder, review schema errors, and try a manual Combine with the same files. Standardize file formats and check the automation's schedule and trigger. |
Related
- Fix a pending batch: map columns when a batch says Pending.
- Add batch info as columns: filter and compare data by batch.
- Add or replace data in a dataset: add a batch, and choose Combine or Replace.
- Keep fewer batches with a retention policy: delete or suspend old batches on a rule.
Something wrong? support@mammoth.io