Skip to content
Try free
Guides › Advanced: open the docs menu

Pull data from a REST API with the CLI

mammoth-cli 2.2.73 and later can connect a REST or Elasticsearch source, sample it, create a dataset from it, and refresh that dataset on a schedule. This guide covers the connection, the sample, the dataset and the refresh. The connector key is generic_rest_api, and every connector command runs in a project (--project PROJECT_ID).

Connections, dataset creation and automations are writes. Confirm each with the person who owns the data before you run it.

mammoth connector get generic_rest_api
mammoth connector connection list generic_rest_api

The first command shows the connection fields and the query keys the connector accepts. The second lists connections you can reuse.

1. Create the connection

Put the settings under config, with flat auth fields: auth_type (none, api_key, bearer or basic), then username and password, or token, or key_name, key_value and inject_into (header or query_param). Don't send a nested auth object. A backend with the fix rejects it with error 4CONN003, and an older backend saves the connection as auth_type: none, so the API answers 401 later.

The base URL must be a public host. Localhost and private addresses are refused.

# request.json (keep it private, mode 0600)
# {"config": {"name": "My API", "base_url": "https://api.example.com/v1", "auth_type": "basic", "username": "u", "password": "p"}}
mammoth connector connection create generic_rest_api --input request.json --yes

The reply holds identity_key. Use it as CONNECTION_KEY below. Error 4SUBS007 means the workspace plan doesn't include this connector.

2. Sample the data

query is a JSON string, not an object. Its keys are endpoint_path, http_method (GET or POST), request_body, query_params, data_root_path (a JSONPath to the records), pagination, flatten_nested_json and max_pages. This step creates no dataset.

# sample.json
# {"query": "{\"endpoint_path\": \"/posts\", \"data_root_path\": \"$\", \"max_pages\": 1}", "validate": false, "data_sample": true}
mammoth connector ds-config create generic_rest_api CONNECTION_KEY --input sample.json --yes

The reply returns real sample rows. Read the column names from data.data_sample.dataschema.

3. Create the dataset with a schedule

schedule_properties and recurrence_info are required. Without them the job fails with "Invalid parameters". on_refresh_action is replace (swap the rows on each refresh) or combine (append). With combine, every refresh adds every row the API returns again, so rows duplicate.

# dataset.json
# {"ds_creation_type": "cloud", "dataset_spec": {"connector_key": "generic_rest_api",
#   "connection_key": "CONNECTION_KEY",
#   "query_properties": {"ds_name": "My data", "table_name": "posts", "query": "<same JSON string as step 2>"},
#   "schedule_properties": {"schedule_type": "moment", "first_pull_at": "now", "on_refresh_action": "replace"},
#   "recurrence_info": {"interval": 1, "start_at": "2027-01-01T02:00:00"}}}
mammoth dataset create --input dataset.json

4. Refresh automatically

# auto.json
# {"description": "Refresh My data", "tasks": [{"task_type": "run_data_retrieval", "details": {"ds_details": [{"ds_id": DATASET_ID}]}}],
#   "conditions": [{"condition_type": "at_specific_time", "details": {"frequency": "daily", "interval": 1, "start_at": "2027-01-01T02:00:00Z"}}]}
mammoth automation create 'Refresh My data' --input auto.json --yes

ds_details must be a list. A start_at less than 2 minutes ahead is rejected as too soon, and a past one starts now. To run the refresh once now, use mammoth automation update AUTOMATION_ID --yes --input '{"patch": [{"op": "command", "path": "run", "value": {}}]}'.

Elasticsearch

Use the same connection with the cluster's address as base_url, and POST on /INDEX/_search. Name the fields you need in _source. A field such as asctime (2026-10-09 15:52:46,290, with a comma before the milliseconds) breaks the load.

  • Documents: "request_body": {"size": 500, "_source": ["@timestamp", "task"]} with "data_root_path": "$.hits.hits[*]._source".
  • Counts per bucket: "request_body": {"size": 0, "aggs": {"c": {"composite": {"size": 10000, "sources": [...]}}}} with "data_root_path": "$.aggregations.c.buckets[*]". The columns are doc_count and key.<source>.

Next up

Report an issue · support@mammoth.io