Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-09-2023 07:11 AM
Having had a quick look, I think your error is because you are trying to add SCD to a STREAMING LIVE table. I believe APPLY CHANGES INTO cannot be used on a streaming table.
You can use a streaming table as a source though.
Simply changing this line:
CREATE OR REFRESH STREAMING LIVE TABLE SCD2_SILVER;to:
CREATE OR REFRESH LIVE TABLE SCD2_SILVER;should be sufficient.
Do make sure you are running a compatible version of databricks. Also, if you want to use Track History, you need to set the pipeline cluster config:
pipelines.enableTrackHistoryto true.
I've also found the databricks SQL parser to give syntax errors sometimes and a little experimenting with removing line breaks, etc. can sometimes help track down errors.