- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-13-2025 02:34 AM
option 2 can be done with TEMPORARY LIVE VIEWs (or TEMPORARY STREAMING TABLE) over a unity catalog table, so not "permanent" I guess.
> for the gold layer is to save that spark SQL code into .py files for each table and import them in the DLT pipeline notebook. Do I loose any DLT features with this approach?
DLT does what you tell it to do, it doesn't necessary care how you tell it. So whether your SQL code is brought to the pipeline by a python script or by *.sql files ... you could have a single huge *.sql file) or SQL cells in a notebook, the outcome should be the same. Note that the python vs SQL api of DLT is not idempotent (some feature currently only available in python afaik)
> Maybe one day I can replace the temporary views with parameterized spark.sql(), when it becomes available to serverless clusters? Or is it possible to use dedicated clusters for DLT?
I think you mean "job cluster"? Nobody should be running DLT (you can't) or any job on personal clusters if that's what you mean.
Also, I'm not entirely sure what you are parametrizing in your views/temp tables, but you could also produce the parametrized SQL code using f-strings in python. Or perhaps you can even leverage forEachBatch (and thus not necessarily DLT!) to run different logic on the same incoming/updated data. if you really insist on DLT/Serverless, you need to think outside of the "dynamic sql" approach in Databricks.