cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

How to writeStream with redshift

rt-slowth
Contributor

I have already checked the documentation below The documentation below does not describe how to write to streaming.
Is there a way to write the gold table (type is streaming table), which is the output of the streaming pipeline of Delta Live Tables in databricks, to redshift in the same way?

 

https://docs.databricks.com/en/connect/external-systems/amazon-redshift.html

2 REPLIES 2

-werners-
Esteemed Contributor III

I doubt Redshift supports streaming writes from Spark.
You could write the changes in the delta table on Databricks (change data feed) into Kafka/Kinesis and consume that in Redshift.  Or overwrite the table.  Or use a merge, but then you need to know what has to be sent to redshift (aka you need to know what was changed/deleted/inserted)

jose_gonzalez
Databricks Employee
Databricks Employee

Only batch processing is supported.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group