DLT Pipeline Failed to create new KafkaAdminClient SQLSTATE: XXKST:

Hanfo2back
New Contributor III

I encountered the error: No LoginModule found for org.apache.kafka.common.security.scram.ScramLoginModule while consuming data from Kafka using a Databricks pipeline. The pipeline had been running smoothly before, but the error appeared on September 11. Upon quick investigation, I noticed that the DLT (Delta Live Table) compute cluster is missing the "scram login" module. I tried installing the Maven library (org.apache.spark:spark-sql-kafka-0-10_2.12:3.5.2) on an all-purpose cluster, which fixed the issue. However, the DLT compute cluster does not allow Maven library installation. How can I resolve this issue?