DLT Pipeline Failed to create new KafkaAdminClient SQLSTATE: XXKST:

[Hanfo2back]

I encountered the error: No LoginModule found for org.apache.kafka.common.security.scram.ScramLoginModule while consuming data from Kafka using a Databricks pipeline. The pipeline had been running smoothly before, but the error appeared on September 11. Upon quick investigation, I noticed that the DLT (Delta Live Table) compute cluster is missing the "scram login" module. I tried installing the Maven library (org.apache.spark:spark-sql-kafka-0-10_2.12:3.5.2) on an all-purpose cluster, which fixed the issue. However, the DLT compute cluster does not allow Maven library installation. How can I resolve this issue?

[ManojkMohan]

Root Cause:

Required Kafka SCRAM login module JAR is missing from the managed DLT cluster environment. This typically happens after Databricks platform updates or cluster environment changes, which may remove previously available libraries from the runtime

Solution / Workaroundss:

Move the Kafka-consuming or -producing step out of DLT into a downstream Databricks notebook or Workflow task on an all-purpose cluster (where required libraries can be installed).

Example: First use DLT for ETL/transformation, write results to a Delta table, then use a separate notebook to read the table and interact with Kafka securely. https://community.databricks.com/t5/data-engineering/delta-live-tables-stream-output-to-kafka/td-p/7...