Amazon MSK integration with Databricks

[Amrit23]

Hello Everyone,

I am a beginner in the world of Data bricks. I am trying to achieve a use case which involves : consuming messages from Amazon MSK and creating a Delta table in Data bricks.

I need to get some insights on what all accesses are supposed to be provided in order to achieve this scenario.The code needs to be written using python.

Please suggest.

Thanks in advance.

[NandiniN]

Hey @Amrit23 ,

Databricks Assistant can help you write the code. You need to use spark.readStream() to access the stream. 

https://docs.databricks.com/en/connect/streaming/kafka.html

And for the Access Permissions:

• Amazon MSK: Ensure you have the necessary permissions to access the Amazon MSK cluster. This typically involves IAM roles and policies that allow you to read from the Kafka topics.
• Databricks: You need to have the necessary permissions to create and manage Delta tables in Databricks. This includes access to the Databricks workspace, clusters, and the ability to write to the storage where the Delta tables will be stored.

References: https://www.databricks.com/blog/2022/08/09/low-latency-streaming-data-pipelines-with-delta-live-tabl...

Thanks!