Write stream to Kafka topic with DLT

[gb]

Hi,

Is it possible to write stream to Kafka topic with Delta Live Table?

I would like to do something like this:

@dlt.view(

name="kafka_pub",

comment="Publish to kafka"

)

def kafka_pub():

return (

dlt.readStream("source_table").selectExpr("to_json (struct (*)) AS value").writeStream.format("kafka").options(**KAFKA_OPTIONS).start()

)

It does not work due to:

RuntimeError: Query defined in function 'kafka_pub' returned 'StreamingQuery'. It must return either a Spark or Koalas DataFrame.

Is there any DLT based solution / workaround?

Thanks.

[shan_chandra]

@gb  - Currently, DLT does not support writing to non-delta sink.  we can divide the pipeline currently into two by writing to a materialized view or delta sink and build a non-DLT job to write out to Kafka sink. Databricks DLT engineering team is working on a feature request to enable DLT to write to kafka sink but we don't have an ETA yet on this. 

[Kaniz_Fatma]

Hey there! Thanks a bunch for being part of our awesome community! 🎉 

We love having you around and appreciate all your questions. Take a moment to check out the responses – you'll find some great info. Your input is valuable, so pick the best solution for you. And remember, if you ever need more help , we're here for you! 

Keep being awesome! 😊🚀