Delta Live Tables error with Kafka SSL

mwoods — Fri, 22 Sep 2023 15:54:44 GMT

We have a spark streaming job that consumes data from a Kafka topic and writes out to delta tables in Unity Catalog.

Looking to refactor it to use Delta Live Tables, but it appears that it is not possible at present to have a DLT Pipeline that can access Unity Catalog, and authenticate with kafka over SSL.

Our code for establishing the readStream is as follows:

df = (spark.readStream .format('kafka') .option('subscribe', _KAFKA_TOPIC) .option('kafka.bootstrap.servers', _KAFKA_BROKER) .option('kafka.security.protocol', 'SSL') .option('kafka.ssl.truststore.location', f'{_AUTH_KAFKA_URL}client.truststore.jks') .option('kafka.ssl.truststore.password', dbutils.secrets.get(scope='auth-secret-scope', key='kafka.ssl.truststore.password')) .option('kafka.ssl.truststore.type', 'JKS') .option('kafka.ssl.keystore.location', f'{_AUTH_KAFKA_URL}client.keystore.p12') .option('kafka.ssl.keystore.password', dbutils.secrets.get(scope='auth-secret-scope', key='kafka.ssl.keystore.password')) .option('kafka.ssl.keystore.type', 'PKCS12') .option('startingOffsets', 'earliest') .load() .withColumn('key', f.col('key').cast(t.StringType())) .withColumn('value', f.col('value').cast(t.StringType())))

But running the pipeline results in the following error:

AnalysisException: [UNSUPPORTED_STREAMING_OPTIONS_PERMISSION_ENFORCED] Streaming options kafka.ssl.truststore.location, kafka.ssl.keystore.location are not supported for data source kafka on a shared cluster.

Is it planned that in future DLT Pipelines might be able to be executed on something other than a shared cluster and write to Unity Catalog? Or to support the kafka.ssl.truststore.location and kafka.ssl.keystore.location options on shared clusters?

Re: Delta Live Tables error with Kafka SSL

Harrison_S — Fri, 06 Oct 2023 00:07:44 GMT

Hello, I believe this is already worked on and should be in the next release for DLT pipelines.

Re: Delta Live Tables error with Kafka SSL

gabriall — Mon, 20 Nov 2023 11:03:38 GMT

Indeed its already patched. you just have to configure your pipeline on the "preview" channel.

topic Delta Live Tables error with Kafka SSL in Data Engineering

Delta Live Tables error with Kafka SSL

Re: Delta Live Tables error with Kafka SSL

Re: Delta Live Tables error with Kafka SSL