Hi,
Is it possible to write stream to Kafka topic with Delta Live Table?
I would like to do something like this:
@dlt.view(
name="kafka_pub",
comment="Publish to kafka"
)
def kafka_pub():
return (
dlt.readStream("source_table").selectExpr("to_json (struct (*)) AS value").writeStream.format("kafka").options(**KAFKA_OPTIONS).start()
)
It does not work due to:
RuntimeError: Query defined in function 'kafka_pub' returned 'StreamingQuery'. It must return either a Spark or Koalas DataFrame.
Is there any DLT based solution / workaround?
Thanks.