Get the latest file from a linked URL dataset
A dataset you create from a URL with Keep it linked keeps following its link. Fetch the latest file with Get the latest in the dataset's panel in the Data Library, or on a schedule with a Dataset Refresh automation. A copy made with Copy it once keeps the link only as a note.
Get the latest file from its link
A dataset created From URL shows where it is linked, such as Linked to example.com. To fetch the file again:
- Open the dataset's panel in the Data Library.
- Click Get the latest ("Fetch the file at its link again.").
Your pipeline replays over the new file. Each refresh follows the dataset's update method in its Settings. A dataset you keep linked when you create it starts on Replace, so fetching an unchanged file again doesn't repeat its rows. If the update method adds rows, the panel notes "Each refresh adds the file's rows again."
Refresh a URL dataset on a schedule
Add the dataset to a Dataset Refresh automation. Dataset Refresh accepts datasets that follow the link they came from.
Two cases can't be refreshed from the link:
- A dataset made from a JSON file: "A dataset made from a JSON file can’t take new rows from its link."
- A copy, which keeps the link as a note only: "A copy. To fetch it again, choose Follow its link in ⋮."
Switch a dataset between linked and copied
From the dataset's ⋮ menu:
| Option | What it does |
|---|---|
| Stop following the link | Turns a linked dataset into a copy. It works like an uploaded file and is never fetched from the link again. |
| Follow its link | Turns a copy back into a linked dataset, so Get the latest and Dataset Refresh fetch the file again. |
If you chose Copy it once but Mammoth couldn't make the copy, it says "The dataset was added, but it still follows its link. Choose Stop following the link in its menu to make it a copy."
See where a dataset came from
The dataset's panel shows a source card: Linked to for a linked dataset, or Copied from for a copy, followed by the link.
Mammoth masks the parts of a link that can carry credentials. A user:password@ login and the values in the query string (for example a token or an X-Amz-Signature) show as ***. Values that only steer how a host serves the file, such as dl, raw, format or gid, stay visible. Everyone who can open the dataset sees the masked link, and refreshes still use the full stored link.
See every dataset that follows a link
The Links group, under the connections on All and Sources, on Data → Connections lists every dataset created from a file at a URL, grouped by website. Each website row shows how many datasets follow it, how many are on a schedule, and Last fetch failed if one failed.
Open a website to see its datasets. Each shows when it was last fetched (such as Fetched 2 hours ago) or why the last fetch failed (for example, "The link answered 404" or "The link could not be reached"), and its schedule or Manual. Click Get the latest to fetch the file again now; the menu offers Automate the refresh (a scheduled refresh) and Open dataset.
When two datasets follow the same link, the second says, for example, same link as Daily Sales, with a note that "Usually one was added by mistake." If a fetch fails, check the URL still returns the file (see Troubleshooting URL imports).
Related
- Import a file from a URL: create the dataset and fix URL errors.
- Refresh a dataset: fetch the file from its link on a schedule.
- Add or replace data in a dataset: how update methods add or replace rows.
- Manage saved connections: the Connections page that holds the Links group.
Something wrong? support@mammoth.io