Collect files from cloud folders into Mammoth
A File Collection automation collects files from folders in Dropbox, Google Cloud Storage, Google Drive, Email (IMAP), SFTP, SharePoint or OneDrive into a folder in Mammoth, on a schedule or when you run it. Create one from Automations > New automation > File Collection, or choose Collect external files into this folder on a folder's menu. It is a plan feature, and your plan can also limit how often it runs.
When to use File Collection
- Pull files that partners or systems drop into cloud storage into Mammoth without manual downloads.
- Feed a folder that a Data Consolidation automation watches, so collected files are merged into one dataset.
Datasets saved once from a file connector can't be refreshed by Dataset Refresh. File Collection followed by Data Consolidation is how you keep bringing in new files from those sources.
Before you start
- Your plan must include File Collection. If it doesn't, the File Collection card shows a plan pill. See What you see when your plan doesn't include a type.
- Your project role needs the Create/manage orchestration flows permission.
- You need a saved connection to the cloud platform, or the details to connect a new one. See Manage saved connections.
Set up a File Collection automation
The wizard has four steps: Select source, Select destination, Set Schedule and Review & save.
Choose the cloud folders to collect from
The Select source step reads "Choose the cloud folder(s) to monitor and collect files from."
- Click Select Cloud Source. The Select Cloud Platform window opens.
- Pick a platform: Dropbox, Google Cloud Storage, Google Drive, Email (IMAP), SFTP, SharePoint or OneDrive.
- Choose a saved connection for that platform, or connect a new one.
- On Select Folders, tick the folders to collect from.
Each source appears as a card with its folder. On the card:
| Option | What it does |
|---|---|
| Switch | Picks a different platform or folder for this source |
| Settings (pencil) | Opens the file options below for this source. Click Done to close them |
| Include subfolders | Also collects files from folders inside the ones you picked |
| Only collect files matching a name pattern | Collects only files whose name Starts with, Ends with, Contains or Matches exactly the text you type (for example Sales) |
| Case sensitive | Makes the name pattern match upper and lower case exactly |
| Close (X) | Removes this source |
To collect from more than one folder or platform, click Add more. One automation can have several sources.
Choose the Mammoth folder for collected files
- In the Select destination step ("Choose a folder inside Mammoth where collected files will be stored."), click Select folder.
- Choose the folder inside Mammoth where collected files are stored. Click Switch to change it.
If you start the automation from a folder's menu (Collect external files into this folder), that folder is already filled in. The first step then shows "Collected files will be stored in" followed by the folder name.
If the chosen folder was deleted, the step asks you to pick another: "Please select a new folder to continue."
Schedule the collection
In the Set Schedule step, choose On schedule or Manual only. For On schedule, pick the unit, Repeat every, First run and an optional End date. Your plan can limit how often a File Collection runs. See Schedules and triggers.
A File Collection runs on its schedule, or manually with Run now.
Review and save the collection
On Review & save, name the automation, add an optional description, check the summary and click Save automation. See Name the automation and save it.
Combine collected files into one dataset
- Set up the File Collection with a Mammoth folder as its destination.
- Set up a Data Consolidation automation that watches that folder and adds its files to a dataset.
In the watched folder's toolbar, the File Collection has its own box with Brings in and Last run. When a collected file doesn't go into the destination dataset, only the Data Consolidation automation's owner is told, because nobody added the file by hand. See Who is told when a file doesn't go in.
Edit a File Collection
Open the automation and click Edit. Under CONFIGURATION you can change the sources, the destination folder and the schedule. See Manage automations.
While a File Collection runs, Monitor's Running tab lists it as File Collection (a cloud storage file retrieval run), with captions such as 12 file(s) collected. See Monitor running and failing work.
Troubleshooting
| Symptom | Cause | Fix |
|---|---|---|
| "Please select a new folder to continue." | The destination folder was deleted | Pick another folder with Select folder |
| Saving fails with "Following frequency not allowed in your plan, please upgrade the plan" | The schedule is more frequent than your plan allows | Pick a coarser unit |
| A scheduled run was skipped | After a plan change, the saved schedule is more frequent than the plan allows | Edit the schedule to a unit your plan allows |
| The File Collection card opens an upgrade window or "Contact Owner to Upgrade" | Your plan doesn't include File Collection | Ask the workspace owner to change the plan |
| Some files weren't collected | A name pattern or Case sensitive excluded them, or they are in subfolders without Include subfolders | Open the source card's Settings and adjust the options |
Related
- Combine incoming data: add collected files to one dataset.
- Schedules and triggers: schedule options and plan limits.
- Create an automation: the steps every type shares.
- Manage saved connections: the cloud connections File Collection uses.
- Folders and folder rules: the Mammoth folders that receive collected files.
Something wrong? support@mammoth.io