Ingest Data into Databricks with Kafka
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-04-2023 07:45 AM
I am trying to ingest data into Databricks with Kafka. I have Kafka installed in a Virtual Machine where I already have the data I need in a Kafka Topic stored as json. In Databricks, I have the following code:
```
df = (spark.readStream .format("kafka") .option("kafka.bootstrap.servers", "<VM_IP:9092>") .option("subscribe", "<topicName>") .load() )
```
Where the printed schema gives me:
```
|-- key: binary (nullable = true) |-- value: binary (nullable = true) |-- topic: string (nullable = true) |-- partition: integer (nullable = true) |-- offset: long (nullable = true) |-- timestamp: timestamp (nullable = true) |-- timestampType: integer (nullable = true)
```
Then I try to write the data to a delta table but the code for that only outputs 'Stream Initializing' and gets stuck there.
I would like to get some help because I cannot figure out what I am doing wrong or missing on this.