topic Re: How to writeStream with redshift in Data Engineering

How to writeStream with redshift

rt-slowth — Mon, 22 Jan 2024 02:13:47 GMT

I have already checked the documentation below The documentation below does not describe how to write to streaming.
Is there a way to write the gold table (type is streaming table), which is the output of the streaming pipeline of Delta Live Tables in databricks, to redshift in the same way?

https://docs.databricks.com/en/connect/external-systems/amazon-redshift.html

Re: How to writeStream with redshift

-werners- — Mon, 22 Jan 2024 15:54:39 GMT

I doubt Redshift supports streaming writes from Spark.
You could write the changes in the delta table on Databricks (change data feed) into Kafka/Kinesis and consume that in Redshift. Or overwrite the table. Or use a merge, but then you need to know what has to be sent to redshift (aka you need to know what was changed/deleted/inserted)

Re: How to writeStream with redshift

jose_gonzalez — Mon, 22 Jan 2024 23:32:05 GMT

Only batch processing is supported.