topic Re: Write stream to Kafka topic with DLT in Get Started Discussions

Write stream to Kafka topic with DLT

gb — Thu, 25 Jan 2024 18:25:17 GMT

Hi,

Is it possible to write stream to Kafka topic with Delta Live Table?

I would like to do something like this:

@dlt.view(

name="kafka_pub",

comment="Publish to kafka"

)

def kafka_pub():

return (

dlt.readStream("source_table").selectExpr("to_json (struct (*)) AS value").writeStream.format("kafka").options(**KAFKA_OPTIONS).start()

)

It does not work due to:

RuntimeError: Query defined in function 'kafka_pub' returned 'StreamingQuery'. It must return either a Spark or Koalas DataFrame.

Is there any DLT based solution / workaround?

Thanks.

Re: Write stream to Kafka topic with DLT

shan_chandra — Thu, 25 Jan 2024 19:59:43 GMT

@gb - Currently, DLT does not support writing to non-delta sink. we can divide the pipeline currently into two by writing to a materialized view or delta sink and build a non-DLT job to write out to Kafka sink. Databricks DLT engineering team is working on a feature request to enable DLT to write to kafka sink but we don't have an ETA yet on this.

Re: Write stream to Kafka topic with DLT

dmytro — Thu, 09 Jan 2025 02:18:47 GMT

@shashas , is a Kafka sink now available? Where can we find information on setting it up, if yes?