Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-01-2024 11:18 AM
Hey @Amrit23 ,
Databricks Assistant can help you write the code. You need to use spark.readStream() to access the stream.
https://docs.databricks.com/en/connect/streaming/kafka.html
And for the Access Permissions:
- Amazon MSK: Ensure you have the necessary permissions to access the Amazon MSK cluster. This typically involves IAM roles and policies that allow you to read from the Kafka topics.
- Databricks: You need to have the necessary permissions to create and manage Delta tables in Databricks. This includes access to the Databricks workspace, clusters, and the ability to write to the storage where the Delta tables will be stored.
References: https://www.databricks.com/blog/2022/08/09/low-latency-streaming-data-pipelines-with-delta-live-tabl...
Thanks!