Skip to content
Docs
Open Mammoth
Learn › Get data in › Connectors: open the docs menu

Connect Apache Hadoop and Hive to Mammoth

The Apache Hadoop connector reads your cluster through HiveServer2, the standard SQL interface for Hadoop, and imports a Hive table or HiveQL query result as a dataset. Open it from Data Library > New dataset > Connectors, search for Hadoop and click Apache Hadoop. Project Admins create the connection; Project Editors can start datasets from a connection the project already has. Which connectors you can add depends on your plan.

What you can bring in from Hadoop

  • Any Hive table the user can read, grouped by schema.
  • The result of your own HiveQL query.

The dataset can refresh daily or only when you ask.

Before you connect Hadoop

You need:

  • A Hadoop cluster running HiveServer2, reachable from the internet.
  • HiveServer2 set up to accept username and password sign-in.
  • A Hive user with SELECT access on the tables you want.
  • The HiveServer2 hostname, port (default 10000), database name, username and password.
  • Mammoth's IP address allowed in your cluster's firewall. See Allow Mammoth through your firewall.

Connect Apache Hadoop to Mammoth

  1. Go to the Data Library, click New dataset, and choose Connectors.
  2. Search for Hadoop and click the Apache Hadoop tile. The connection form opens.
  3. Fill in the fields below and click Connect. If the check fails, the error appears at the bottom of the form.
  4. Once Mammoth has connected, the connector workspace opens on your tables. See Connect a database or app for the shared steps.
FieldWhat to enterExample
Host URLHostname or IP address of your HiveServer2 nodehive.cluster.example.com
PortHiveServer2 TCP port10000 (default)
UsernameHive user with SELECT accessmammoth_reader
PasswordPassword for the Hive user
DatabaseName of the Hive databaseanalytics
Display NameOptional label for this connectionHadoop Cluster Prod

Query a Hive table with HiveQL

  1. In Tables, click a table. Mammoth opens its rows with the SELECT * query that read them, shown above the rows.
  2. To write HiveQL instead, click Edit SQL, edit the query, and click Run.
SELECT order_id, customer_id, amount, order_date
FROM analytics.orders
WHERE order_date >= '2024-01-01'
LIMIT 10000

AI query writing (Write SQL) isn't available for Hadoop. Connect a database or app covers the full SQL view.

Save Hadoop data as a dataset

  1. Click Save as dataset.
  2. Enter a Name and pick a folder.
  3. Under Keep it up to date, choose Daily at 06:00 or Only when I ask.
  4. Next to the sentence that says what each refresh does, click Change and pick an option under On each refresh (see the table below).
  5. Click Save as dataset.
On each refreshWhat it does
Replace all rowsMatches the source after every refresh
Add only new rowsKeeps what you have and adds rows above the last value of a Key column
Add every row againShown instead when the result has no numeric or date column, so each refresh adds a full copy
Keep only the newest rowsEmpties the dataset, then brings in rows since the last refresh (needs a Key column)

A key column is a column whose values only ever go up, such as an ID or a date.

Refresh Hadoop data on a schedule

Choosing Daily at 06:00 creates a Dataset Refresh automation that you can reschedule later. After the dataset is saved, you can change what a refresh does in its Update method: Replace all data, Add new rows only or Keep only the newest rows.

Troubleshoot Hadoop connections and slow Hive queries

SymptomLikely causeFix
Connection failsPort 10000 is closed to MammothOpen port 10000 inbound from Mammoth's IP address
Connection failsThe user can't read the databaseRun a test query in Beeline from your cluster to check the user's access
Queries are slowLarge Hive scans take timeUse LIMIT while you explore, then remove it before you save so the full result is imported
Was this page helpful?

Something wrong? support@mammoth.io