cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

How to writeStream with redshift

rt-slowth
Contributor

I have already checked the documentation below The documentation below does not describe how to write to streaming.
Is there a way to write the gold table (type is streaming table), which is the output of the streaming pipeline of Delta Live Tables in databricks, to redshift in the same way?

 

https://docs.databricks.com/en/connect/external-systems/amazon-redshift.html

2 REPLIES 2

-werners-
Esteemed Contributor III

I doubt Redshift supports streaming writes from Spark.
You could write the changes in the delta table on Databricks (change data feed) into Kafka/Kinesis and consume that in Redshift.  Or overwrite the table.  Or use a merge, but then you need to know what has to be sent to redshift (aka you need to know what was changed/deleted/inserted)

jose_gonzalez
Moderator
Moderator

Only batch processing is supported.

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.