Import Apache Kafka topic messages into Mammoth
The Apache Kafka connector reads messages from a topic on your Kafka cluster and saves them in Mammoth as a dataset. Open it from Data Library > New dataset > Connectors and search for Apache Kafka. Project Admins create the connection; Project Editors can start datasets from a connection the project already has. Which connectors you can add depends on your plan.
What you can bring in from Kafka
- Messages from one topic, picked from the topics Mammoth finds on the cluster.
- Each message's value, plus columns for Kafka Offset, Kafka Partition, Kafka Timestamp and Kafka Topic.
Before you connect Kafka
You need:
- The cluster's bootstrap servers, reachable from the internet.
- Its security protocol, and a SASL username and password if it uses SASL.
- Mammoth's IP address allowed through your firewall for every bootstrap server. The connection form shows the address. See Allow Mammoth through your firewall.
Connect Apache Kafka to Mammoth
- Go to the Data Library, click New dataset, and choose Connectors.
- Search for Apache Kafka and click its tile. The connector workspace opens with the Chat showing. See Connect a database or app for the shared steps.
- Enter the connection details below and connect.
| Field | What to enter |
|---|---|
| Bootstrap Servers | The broker addresses, comma-separated, such as broker1:9092,broker2:9092 |
| Security Protocol | Default PLAINTEXT. Use the protocol your cluster requires (see below) |
| Username (SASL only) | SASL username, if your cluster uses SASL |
| Password (SASL only) | SASL password, if your cluster uses SASL |
| Display Name | Optional label for this connection in Mammoth |
| Security protocol | When to use it |
|---|---|
PLAINTEXT | The cluster needs no sign-in |
SASL_PLAINTEXT | Username and password sign-in without TLS |
SASL_SSL | Username and password sign-in over TLS |
With either SASL protocol, Mammoth signs in with the SASL PLAIN mechanism.
Choose the Kafka topic, offset and message limit
The Kafka connector's own setup form asks for these settings:
| Setting | What it does | Default |
|---|---|---|
| Topic | The topic to read, picked from the topics Mammoth finds on the cluster | |
| Consumer Group ID | The consumer group Mammoth reads as | mammoth-consumer |
| Offset Reset | Earliest (all messages) reads from the start of the topic; Latest (new messages only) reads only new messages | Earliest (all messages) |
| Max Messages Per Run | The most messages Mammoth reads in one run | 10000 |
In the connector workspace, you can tell the chat which topic you want.
Refresh Kafka data and choose an update method
After you save, a Kafka dataset's Update method offers two options:
| Update method | What it does |
|---|---|
| Replace all data | Rebuilds the dataset from the messages read on every refresh |
| Add to existing data | Adds the messages read on each refresh to the rows you already have |
Kafka refreshes don't use a key column. To read new messages on a schedule, use a Dataset Refresh automation.
Kafka limits
- One run reads at most the Max Messages Per Run value (default
10000).
Troubleshoot Kafka connections
| Symptom | Likely cause | Fix |
|---|---|---|
| "Failed to connect to Kafka broker. Please check your connection details." | Mammoth can't reach a broker | Check Mammoth's IP address can reach every bootstrap server, and the addresses are right |
| "Authentication failed. Please check your username and password." | SASL details don't match | Check Security Protocol and the SASL username and password match the cluster's settings |
| A message says the topic wasn't found and lists available topics | The topic name is wrong | Pick a topic from the list Mammoth shows |
Related
- MQTT: collect messages from an MQTT broker.
- Receive data by webhook: let another system push data to Mammoth.
- Connect a database or app: the shared connector workspace steps.
- Allow Mammoth through your firewall: the IP address to allow.
- Refresh a dataset: change when the dataset refreshes.
Something wrong? support@mammoth.io