Amazon MSK integration with Databricks

Amrit23 — Sat, 26 Oct 2024 15:53:48 GMT

Hello Everyone,

I am a beginner in the world of Data bricks. I am trying to achieve a use case which involves : consuming messages from Amazon MSK and creating a Delta table in Data bricks.

I need to get some insights on what all accesses are supposed to be provided in order to achieve this scenario.The code needs to be written using python.

Please suggest.

Thanks in advance.

Re: Amazon MSK integration with Databricks

NandiniN — Fri, 01 Nov 2024 18:18:26 GMT

Hey @Amrit23 ,

Databricks Assistant can help you write the code. You need to use spark.readStream() to access the stream.

https://docs.databricks.com/en/connect/streaming/kafka.html

And for the Access Permissions:

Amazon MSK: Ensure you have the necessary permissions to access the Amazon MSK cluster. This typically involves IAM roles and policies that allow you to read from the Kafka topics.
Databricks: You need to have the necessary permissions to create and manage Delta tables in Databricks. This includes access to the Databricks workspace, clusters, and the ability to write to the storage where the Delta tables will be stored.

References: https://www.databricks.com/blog/2022/08/09/low-latency-streaming-data-pipelines-with-delta-live-tables-and-apache-kafka.html

Thanks!

topic Amazon MSK integration with Databricks in Get Started Discussions

Amazon MSK integration with Databricks

Re: Amazon MSK integration with Databricks