Hi ! 

Ensure your code is set up to use these libraries. Here is the complete example:

•   Navigate to your cluster configuration:

• Go to your Databricks workspace.
• Click on "Clusters" and select your cluster.
• Go to the "Libraries" tab.

•   Install the necessary Maven libraries:

• Click on "Install New".
• Choose "Maven" as the library source.
• Add the following Maven coordinates:
  • org.apache.kafka:kafka-clients:2.8.0
  • org.apache.kafka:kafka_2.13:2.8.0

df1 = df.selectExpr("CAST(null AS STRING) as key", "to_json(struct(*)) AS value")

df1.writeStream \
    .format("kafka") \
    .option("kafka.bootstrap.servers", "your_broker_details") \
    .option("kafka.security.protocol", "SASL_SSL") \
    .option("kafka.sasl.mechanism", "SCRAM-SHA-512") \
    .option("kafka.sasl.jaas.config", "org.apache.kafka.common.security.scram.ScramLoginModule required username='your_username' password='your_password';") \
    .option("topic", "your_topic1") \
    .option("checkpointLocation", "/path/to/your/checkpoint") \
    .option("kafka.metadata.max.age.ms", "120000") \
    .start()

 Mehdi TAJMOUATI

WyTaSoft

medium

linkedin