- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-10-2025 02:06 AM
Hey adriennn, thanks for the long answer.
So 1 is just not an optoin. the goal is to do a whole migration.
Option 2 would mean that all those temporary tables/views I make to create a gold table would become permanent through the use of LIVE.temp1, wouldnt they?
Option 3 is basically what I am trying to do, migrating an on-prem DWH into a databricks Lakehouse. Bronze and Silver layers are already successfully migrated into databricks. Also I already translated all the gold TSQL to spark SQl and split the domains/data sources into their own DLT pipelines.
Now my best guess (I am new to databricks) for the gold layer is to save that spark SQL (with temporary views instead of parameterized spark.sql()) code into .py files for each table and import them in the DLT pipeline notebook. Do I loose any DLT features with this approach?
Maybe one day I can replace the temporary views with parameterized spark.sql(), when it becomes available to serverless clusters? Or is it possible to use dedicated clusters for DLT?