Skip to content
Try free
Learn › Data Library: open the docs menu

Combine datasets

Combine puts the rows of two or more datasets together in one place. It is for anyone who maintains datasets, for example monthly exports that arrive as separate files, or the same report from several regions. You choose one of two ways, and they leave you with different libraries. Start with Opening Combine.

To combine columns side by side (matching rows on a key) instead of stacking rows, use a Join or Lookup in a View's pipeline.

Opening Combine

  1. In the Data Library, tick two or more datasets in the list.
  2. In the selection bar, click Combine. You can also right-click one of the selected rows and choose Combine.
  3. The Combine N datasets dialog opens: "Put their rows together in one place. Two ways, and they leave you with different libraries."

Combine is available only when the selection holds datasets that are ready:

MessageWhy
"Only datasets can be combined."The selection includes a folder, a View or a raw file
"“Sales” is not ready yet."One of the datasets is waiting on your input
"“Sales” is still processing."One of the datasets is still processing

You need edit access to the project. With view-only access, Combine and the other selection actions are unavailable.

Choose how to combine

Step 1 shows two cards, each with a before-and-after diagram. When both are available, nothing is selected for you; when only one is, it is picked. Pick one and click Next.

OptionWhat happensAfterwards
Batches in one dataset"The originals are consumed, and their refreshes go with them." The rows become batches in the dataset you keep.N datasets become 1
Send into a dataset"The originals stay, and keep sending rows as they refresh." Their rows append into a new or existing dataset.N datasets become N+1 (or stay N when you send into one you already have)

Pick Batches in one dataset to tidy up: for example, two uploads of the same report that should have been one dataset. Pick Send into a dataset when the originals have their own refreshes or pipelines you want to keep, and you want one dataset that collects from all of them.

Batches in one dataset

This option is available only when all of these are true. When it is not, its card says why.

ConditionMessage when it fails
Exactly two datasets are selected"Two datasets at a time" (the column mapping is set up per pair)
Neither dataset has more than one View"“Sales” has 2 views, so there is no answer to which one survives."
Neither dataset has steps in its View"“Sales” has 3 steps built on it, so its rows cannot move."

Steps

  1. Choose Batches in one dataset and click Next.
  2. Under Keep which one?, pick the dataset that stays. Each option shows its columns and rows. "Each batch keeps its own name and date, so you can suspend or remove one later."
  3. If the two datasets have different numbers of columns, a warning says you will match columns next. Otherwise, any columns that don't line up are matched in the next step.
  4. Click Combine into (the button names the dataset you keep).
  5. The Add data dialog opens on the dataset you kept, with Select existing and the other dataset already chosen. Finish there: check the column match, and leave Delete dataset after combining ticked so the other dataset stops existing on its own. See Path B: select an existing dataset.

Expected result: The dataset you kept holds both sets of rows as separate batches. Check them in its Batches section under Settings & history.

Send into a dataset

This option works for any number of datasets. Each original gets a Send to Dataset step at the end of its View (the View you were last in, or its first View). The step runs every time that View's pipeline runs, so the destination keeps receiving rows as the originals refresh.

Send into a new dataset

  1. Choose Send into a dataset and click Next.
  2. Under Send into, choose New dataset ("The originals stay and keep sending to it.").
  3. Type a name in Name the new dataset. The new dataset goes into the folder you are in, or the top of the project. If the name is already used there, you see "There’s already a dataset called this here."
  4. Check the column review (see Review columns).
  5. Click Create (the button shows the new name), or press Enter in the name box.

The dialog stays open and shows its progress: "Reading the columns of each dataset…", "Creating the new dataset…", then "Sent 2 of 3…".

Send into an existing dataset

  1. Choose Send into a dataset and click Next.
  2. Under Send into, choose An existing dataset ("Any dataset in this project. Its columns stay as they are.").
  3. In the finder that opens, pick the destination and click Choose a dataset. You can pick any dataset in the current project except the ones you are combining. To pick a different one, click Change.
  4. Check the column review.
  5. Click Send into (the button names the destination).

If one of the originals already sends its rows to that dataset, it is left out of this send, and the dialog says so. Sending it twice would add its rows twice on every refresh.

Review columns

Before you send, Review columns shows what happens to each column, with the counts (for example 4 in all, 1 in some).

For a new dataset, the new dataset gets every column from every original, matched by name:

  • A column that only some originals have is kept, and is "Empty in rows from" the others.
  • Columns with close names are flagged ("Looks like the same column as …"). Click Make one column to merge them.
  • Leave out drops a column from the new dataset.
  • A column that is not the same type in every dataset becomes text, and the review says which types it found.

For an existing dataset, its columns stay as they are. Each original's columns are matched into the destination's columns by name, and you can send a column into a different destination column. Columns with no match are listed under "Left out: these have no column in" the destination. If nothing from one original fits, the button is disabled until you send at least one of its columns into a column in the table, or go back and leave that dataset out of the selection.

What you see afterwards

ResultMessage
New dataset created"Created “All regions”. Its rows come from 3 datasets and update when they refresh." with Open
Sent into an existing dataset"“All regions” now gets rows from 2 more datasets and updates when they refresh."
Some columns had no match"Region, Code had no column of the same name in “All regions” and were not sent."
A column was widened"Order ID held more than one kind of value, so it is text in the new dataset."
Nothing sent"Nothing was sent. Your datasets are unchanged."

Sources are sent one at a time, and the send stops at the first failure. Then the message says how many were sent, for example "Sent 2 of 3 to “All regions”, then West failed." The destination keeps what was sent.

Mammoth checks your workspace's storage limit before it sends. If the workspace is over its limit, the send does not start.

Report an issue · support@mammoth.io