Fix files a watched folder didn't add to a dataset
A watched folder is a Data Library folder that a Data Consolidation automation takes files from. A file that can't be added stays in the folder with a chip and a one-line reason, and an action such as Add anyway, Match the columns or Review on its row. The person who added it and the automation's owner are notified, and the automation shows Files not added until someone deals with it.
To set up the watched folder itself, see Combine files dropped in a folder into one dataset.
Read a watched folder in the Data Library
The folder's automations box
A watched folder's toolbar shows a box for each automation on it, after New folder. A rule's box reads like Files go into Sales, or *_sales.csv into Sales when it has a name pattern. A suspended automation shows Paused. Click a box to open its card:
| Field | What it shows |
|---|---|
| Status | Active or Paused |
| Takes | Which files the rule takes, for example "names ending “_sales.csv” · includes subfolders", or "all files added here" |
| Goes into | The destination dataset and where it lives, with a link to its folder |
| Last file | When the last file arrived, or "no files yet" |
The card ends with: "Once a file is added it leaves this folder. A file that can't be added stays here, and we'll tell you why." Click View to open the automation in the Automations tab. A File Collection that brings files into the folder has its own box with Brings in and Last run.
With two or more automations, a list box (for example 2 automations) opens This folder's automations. With more than two, only the first keeps its own box and the rest fold into the list box, which then reads, for example, +2 more. The list shows the rules in order (under "Files added here go by the first rule they match"), then Bringing files in. The last line says what happens to the rest: "Anything no rule takes stays here", or "Nothing stays here: rule 2 takes every file the ones above it don't" when a rule takes all files.
Upload into a watched folder
When you upload files into a watched folder from Add data, the upload shows Where these files go while it checks ("Checking where these files go…"). Each file shows its destination (for example "→ Sales"), or Stays here with "No rule takes it, so it becomes its own dataset here." A file matching two rules says which wins, for example "Also matches Returns, but Sales comes first." The note reads "Added files leave this folder once they're in their dataset."
Folder, dataset and file chips
In the Data Library list, a watched folder shows Watched and which dataset it feeds ("feeds Sales" or "feeds 2 datasets"). A destination dataset shows "Fed by" and the folder's name. While files move, the folder shows Adding 3 files… and the destination shows Adding…. When files wait on someone, the folder shows, for example, 2 not added.
Each file still in a watched folder shows a chip and a one-line reason:
| Chip | What it means | Example reason |
|---|---|---|
| Reading… | The file is being read, for example with the destination's recipe | "Reading with Sales's recipe · you can leave, we'll tell you if it doesn't go in" |
| Adding… | The file is on its way into the destination | "Adding to Sales…" |
| Not added | The file didn't fit the destination and is waiting on you | "Its columns don't match Sales" |
| Didn't go in | Something went wrong while adding it | "Couldn't be added to Sales" or "It stopped on its way to Sales; add it again" |
| Kept separate | The destination's recipe couldn't read it, so it stays as its own dataset. It also shows Meant for Sales | "Sales's recipe couldn't read it, so it was kept as its own dataset" |
| Duplicate | Same contents as a file already added, so it was skipped | "Same contents as sales_jan.csv, added 2 days ago. Skipped." |
| Needs password | The file is password protected | "Waiting for its password, then it goes to Sales" |
| Pick sheets | A workbook with several sheets is waiting for you to choose | "Pick which sheets go into Sales" |
A file that stays "moving" for more than 2 hours without an answer is shown as Didn't go in with "It stopped on its way to Sales; add it again".
Fix a file that didn't go into the dataset
Each waiting file has an action on its row:
| Action | Shown when | What it does |
|---|---|---|
| Add anyway | The file was skipped as a Duplicate | Adds it after all. You see "Adding sales_jan.csv to Sales", then "Added sales_jan.csv to Sales" |
| Match the columns | The file is Not added because of new columns, a changed type or columns that don't match | Opens the destination with the waiting file's column mapping, so you can map its columns in. If the destination is gone you see "Couldn't find Sales. It may have been moved to trash." |
| Pick sheets | A workbook is waiting for you to choose its sheets | Opens the file so you can choose which sheets go in |
| Review | Any other waiting file: Kept separate, Needs password, or Not added for another reason | Opens the file so you can fix it (enter its password, read it in the interpreter, or check it) |
You can also correct the file and upload it again. Add anyway is also on the file in the automation's History.
Add PDFs and messy spreadsheets with a recipe
If the destination dataset was made by reading a PDF or a messy spreadsheet in Add data, it has a recipe. New files in the folder are read with that recipe, then added. The type picker points to this path: "Reading a spreadsheet or PDF? Start from Add data, and the automation comes with it." See Extract tables from PDFs.
When the recipe can't be used, the file is not added:
| Reason shown on the file | What to do |
|---|---|
| "Sales's recipe couldn't read it, so it was kept as its own dataset" | It stays as its own dataset, marked Kept separate and Meant for Sales. Click Review |
| "No table in it matches Sales's recipe, so it was kept as its own dataset" | Check the PDF is the same kind of document as the earlier ones |
| "Sales has no recipe yet, so it was kept as its own dataset" | Read a file once in the interpreter to create the recipe |
| "Read it once in the interpreter, so Sales can take files like it" | Click Review and read the file in the interpreter |
| "Not added to Sales yet: the first one waits for you to check it was read right" | The first file read with a new recipe waits for your check. Click Review |
| "Not added to Sales: its figures don’t add up" | The read figures don't add up to the total the recipe expects. Check the file against the document |
| "Not read: this month's AI pages for PDFs are used up" | Wait for next month's AI pages, then add the file again |
| "Not read: PDFs need an AI key, and none is set up" | Ask your admin to set up the AI key |
| "Its rule names several datasets, and a PDF can go to one" | Give the PDF its own rule with a single destination |
When a PDF or image batch couldn't be matched to the recipe, the automation's status shows Needs your input until someone deals with it.
Who is told when a file doesn't go in
When a file doesn't go in or is kept separate, the person who added it and the automation's owner get a notification. It comes once per run for each destination, naming the first file and counting the rest, so 50 files with the wrong columns are one notice. For a file nobody added by hand (for example one collected by a File Collection or moved in by an automation), only the owner is told. Everyone else in the project still sees the file's chip and the automation's History.
These files also show in Monitor under Needs Attention (in an Automations group when other items are listed too), as File didn't go in or File kept separate, with Review. A file whose columns didn't fit shows there as a pending item for its destination instead. Duplicates raise no notice: they show only in the folder and History.
Troubleshooting
| Symptom | Cause | Fix |
|---|---|---|
| Status shows Files not added after an upload | A file didn't go in, for example because its columns or types don't match. A duplicate doesn't change the status: it shows a Duplicate chip | Find the file with the Not added, Didn't go in or Kept separate chip. For a column mismatch, click Match the columns. For Kept separate, Needs password or another Not added file, click Review. A Didn't go in file has no action: correct it if needed and upload it again |
| A file stayed in the folder with no chip | No rule took it: its name matches no pattern, or the automation is suspended | Check the rules in the folder's automations box. Make sure the automation is Active and the file name matches |
| A file went to the wrong dataset | It matched more than one rule, and an earlier rule took it | Narrow the earlier rule's pattern, or reorder the rules. Files already added stay where they are |
| Match the columns says "Couldn't find Sales. It may have been moved to trash." | The destination dataset was moved to Trash | Restore it from Trash, or edit the automation to point to another dataset |
Related
- Combine incoming data: set up the watched folder and its rules.
- Manage automations: read a Data Consolidation's History.
- Notifications: the notice you get when a file doesn't go in.
- Monitor running, failing and paused pipelines: files waiting in Needs Attention.
- Extract tables from PDFs: create the recipe that reads new PDFs.
Something wrong? support@mammoth.io