Skip to content
Learn › Get data in: open the docs menu

Batches in Mammoth: see, suspend and delete what arrived

A batch is one load of data into a dataset: an uploaded file, a connector refresh or a consolidated file. Each batch keeps its name, date, row count and source, so you can see what arrived, take a batch out with Suspend, or remove it with Delete. Open the dataset's panel in the Data Library, then Settings & history, then Batches.

Dataset panel on Settings & history with the Batches section open: two batches, each with name, date, row count, source and Active state, plus Add Batch Info to Dataset, Set up retention policy and Add data

How batches make up a dataset

One dataset can hold many batches, each a separate source or time period. All batches share the same columns, Views show all batches combined by default, and pipeline steps apply to every batch.

Sales Dataset
├── April_Sales_North.csv (127 rows) - Batch 1
├── April_Sales_South.csv (143 rows) - Batch 2  
└── June_Sales_Combined.csv (298 rows) - Batch 3
    Total Dataset: 568 rows across 3 batches

Batches give you an audit trail (what came from where and when), versions you can roll back or compare without separate backups, and analysis by batch or source inside one dataset.

Where batches come from

Batches are created for you. Whether a new load adds a batch or replaces the old ones depends on the dataset's Update method (see Choose add or replace).

SourceWhat it adds
File upload or Add dataOne batch per file, named after the file, with the time captured
Data Consolidation automationOne batch per new file in a monitored Data Library folder (not an external folder), named from the file name, after a schema check
Dataset Refresh automationPulls the latest data from a connector on a schedule (daily, weekly or monthly). In Combine mode, a new batch per refresh, keeping history. In Replace mode, the previous batches are deleted and one new batch is added; only their details stay in the Activity Log.

To set up Dataset Refresh: go to Automations, then Dataset Refresh; select the dataset(s) with a live connection; set the schedule (frequency, weekday, start and end date); and for several datasets choose Sequential or Parallel. Sequential lets you drag to set the order.

To set up Data Consolidation: go to Automations, then Data Consolidation; select a source folder to monitor; select the destination dataset (optionally filtered by file name pattern); then review and save. Combine or Replace is not chosen here: it is the destination dataset's Update method, which you can change at any time. When a file lands, Mammoth checks its columns, adds it as a new batch, runs the pipeline, and delivers exports to their destinations.

The Activity Log records every batch operation, manual or automated: additions, suspend and unsuspend, deletions and data retention runs. See Activity log.

Read the batch list

The list shows 10 batches per page; use the previous and next arrows to move between pages.

FieldWhat it shows
NameThe source file name or a custom name. Select it to preview the batch.
DateThe exact creation date and time
Row CountRows this batch contributes
SourceAn icon for where the batch came from: a file upload, a database connector, or a Combine Dataset operation
StateActive, Suspended, or Pending Delete (waiting for approval under a retention policy)
ActionWhat it does
DeletePermanently deletes the selected batches. This can't be undone.
Suspend / UnsuspendTakes batches out of use, or brings them back, without deleting them
Add dataA button below the list (file-based datasets only) that opens the Add data dialog
Add data dialog for batch1.csv with Upload file selected, Update method Add to existing data, the drop zone, Column mapping: Default, and a disabled Submit button Batches section with both batches ticked, showing Suspend / Unsuspend and Delete, and the tooltip "Remove or restore a specific data batch from all associated views."

A batch that reads Applying recipe or Pending is still being added: see Fix a pending batch.

Suspend a batch or delete it

Suspend removes a batch from Views and pipeline runs straight away, but keeps all its data and details. Select it again and choose Suspend / Unsuspend to bring it back. Use it to test the impact of a batch, quarantine bad data, leave out a season, or prepare a rollback.

Delete removes a batch permanently after you confirm ("Are you sure you want to delete the selected batches?"). Use it only when the data is no longer needed for analysis or compliance. A retention policy can also delete batches for you.

PropertySuspendDelete
Data keptYesNo
Details keptYesYes
RecoveryInstant, with one clickNot recoverable in the app. Batches removed by a retention policy are also permanent; their details stay in the Activity Log
Best useTemporary exclusion, testingPermanent exclusion

Views that don't update after a new batch

Auto-sync is a setting on each View's incoming link. When it is off, the View is not rerun when its dataset gets a new batch, and the new data stays pending. A warning triangle (⚠) with "Data may be inconsistent" appears in the dataset's Views list, and Monitor's Needs Attention tab shows "new batch received, auto-sync off" with Preview and Process buttons. See Views: create, rename and delete.

Keep batches easy to work with

  • Use Replace for latest-only data, so batches don't pile up.
  • Set up a retention policy to delete or suspend old batches by age, count or condition.
  • Name files before upload, because batch names come from file names. Sales_North_Region_Q2_2024.csv is descriptive and sortable; data_export_final_v3.csv has no context or date. A pattern such as Location_YYYY_MM_DD.csv makes batches easy to filter.
  • Apply a Conditional Filter early in pipelines on large datasets to reduce processing.
LimitValue
Batch size from a file uploadUp to 50 MB per file (see Limits and quotas)
Batches per page in the list10

Troubleshooting batches

SymptomCauseFix
Columns don't line up when combining ("Schema mismatch: column mapping required")Column names differ (case and spacing count), types are incompatible, or columns are missingCheck names exactly, map columns in the pending batch's Column Mapping tab, and use the same export template for every source
Unexpected row countsCombine or Replace wasn't what you expected, or batches duplicate each otherCheck the dataset's update method and each batch's row count
A batch isn't in the analysisIt is suspended, or transformations didn't rerunCheck it shows Active, and that the pipeline reran after the batch was added
Historical data is missing after adding a batchReplace was selected (can't be undone), batches were suspended, or a retention policy removed themCheck suspended batches and the retention policy, and review the Activity Log for removals
Data doesn't look updated after ReplaceYour browser shows an old copyHard-refresh (Ctrl+Shift+R on Windows, Cmd+Shift+R on Mac) and check the Activity Log shows the Replace completed
A suspended batch still shows in a ViewThe status hasn't refreshedCheck the batch shows as suspended, close and reopen the dataset, and clear cached pipeline results. If it still appears, contact support.
Data Consolidation doesn't combine filesThe file landed in the wrong place, failed the schema check, or the automation didn't run. A file with different columns is added as a separate file.Check the Activity Log for the run, confirm the folder, review schema errors, and try a manual Combine with the same files. Standardize file formats and check the automation's schedule and trigger.
Was this page helpful?

Something wrong? support@mammoth.io