Write stream to Kafka topic with DLT

gb · ‎01-25-2024

Hi,

Is it possible to write stream to Kafka topic with Delta Live Table?

I would like to do something like this:

@dlt.view(

name="kafka_pub",

comment="Publish to kafka"

)

def kafka_pub():

return (

dlt.readStream("source_table").selectExpr("to_json (struct (*)) AS value").writeStream.format("kafka").options(**KAFKA_OPTIONS).start()

)

It does not work due to:

RuntimeError: Query defined in function 'kafka_pub' returned 'StreamingQuery'. It must return either a Spark or Koalas DataFrame.

Is there any DLT based solution / workaround?

Thanks.

shan_chandra · ‎01-25-2024

@gb - Currently, DLT does not support writing to non-delta sink. we can divide the pipeline currently into two by writing to a materialized view or delta sink and build a non-DLT job to write out to Kafka sink. Databricks DLT engineering team is working on a feature request to enable DLT to write to kafka sink but we don't have an ETA yet on this.

dmytro · ‎01-08-2025

@shashas , is a Kafka sink now available? Where can we find information on setting it up, if yes?