Is there support for Confluent Kafka in databricks free edition

[Suresh_Ulhasnag]

I want to do a streaming POC using trial version of Confluent Kafka.

Will databricks free edition support it?

[AbhaySingh]

You CAN use Confluent Kafka in Databricks Free Edition, but only for micro-batch processing (not continuous real-time streaming). If you need true real-time continuous streaming with ProcessingTime triggers, you'll need a paid plan with classic compute.

[Pat · Accepted solution]

Hi @Suresh_Ulhasnag ,

Yes it's supported. You can have a look at my github repo: https://github.com/cloud-data-engineer/data/blob/main/dlt_telco/docs/README.md

In bronze layer I am connecting to Confluent Kafka: https://github.com/cloud-data-engineer/data/blob/main/dlt_telco/src/bronze.py
You can see example setup there, i.e.:

# Configure Kafka security options
KAFKA_SECURITY_OPTIONS = {
    "kafka.security.protocol": "SASL_SSL",
    "kafka.sasl.mechanism": "PLAIN",
    "kafka.sasl.jaas.config": f"org.apache.kafka.common.security.plain.PlainLoginModule required username='{api_key}' password='{api_secret}';",
}