NandiniN
Databricks Employee
Databricks Employee

Hey @Amrit23 ,

Databricks Assistant can help you write the code. You need to use spark.readStream() to access the stream. 

https://docs.databricks.com/en/connect/streaming/kafka.html

And for the Access Permissions:

  • Amazon MSK: Ensure you have the necessary permissions to access the Amazon MSK cluster. This typically involves IAM roles and policies that allow you to read from the Kafka topics.
  • Databricks: You need to have the necessary permissions to create and manage Delta tables in Databricks. This includes access to the Databricks workspace, clusters, and the ability to write to the storage where the Delta tables will be stored.

References: https://www.databricks.com/blog/2022/08/09/low-latency-streaming-data-pipelines-with-delta-live-tabl...

Thanks!