cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Failed to create new KafkaAdminClient

FilipezAR
New Contributor

I want to create connections to kafka with spark.readStream using the following parameters:

 

kafkaParams = {
    "kafka.sasl.jaas.config": f'org.apache.kafka.common.security.plain.PlainLoginModule required username="{kafkaUsername}" password="{kafkaPassword}";',
    "kafka.sasl.mechanism": "PLAIN",
    "kafka.security.protocol": "SASL_PLAINTEXT",
    "kafka.bootstrap.servers": kafkaBootstrapServers,
    "subscribe": kafkaTopic,
    "group.id": kafkaGroupID
}

 

I get the following error: kafkashaded.org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient

The problem is that it doesn't see: org.apache.kafka.common.security.plain.PlainLoginModule

But, if I use the library for python (confluent_kafka) everything works correctly with the same parameters inside Notebook Databricks.

Maybe someone has an idea about this?

1 REPLY 1

Kaniz_Fatma
Community Manager
Community Manager

Hi @FilipezAR,

If none of these suggestions work, you might want to consider using the Confluent Kafka Python library, which you mentioned is working correctly. 

Remember to replace all placeholders like {kafkaUsername}, {kafkaPassword}, kafkaBootstrapServers, kafkaTopic, and kafkaGroupID with actual values. If you’re already doing this, please ignore this reminder.

Please let me know if you need further assistance! 😊

 
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!