How to writeStream with redshift
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-21-2024 06:13 PM
I have already checked the documentation below The documentation below does not describe how to write to streaming.
Is there a way to write the gold table (type is streaming table), which is the output of the streaming pipeline of Delta Live Tables in databricks, to redshift in the same way?
https://docs.databricks.com/en/connect/external-systems/amazon-redshift.html
- Labels:
-
Workflows
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-22-2024 07:54 AM
I doubt Redshift supports streaming writes from Spark.
You could write the changes in the delta table on Databricks (change data feed) into Kafka/Kinesis and consume that in Redshift. Or overwrite the table. Or use a merge, but then you need to know what has to be sent to redshift (aka you need to know what was changed/deleted/inserted)
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-22-2024 03:32 PM
Only batch processing is supported.
