How to create Delta live tables in Silver layer
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-08-2024 02:35 AM
How to create Delta live tables in Silver layer
Hi DB Experts,
Having basic questions :
I am working on Madalian Architecture (B, S, G) Layers.
on B i am getting Delta files (Parq) format. with log folders. One folder for one table, multiple files are getting placed after certain size... But I dont see any Detla tables.
Now from these files I want to create straming table in "Silver" Layer... I am unable to find solution. Following example shows to create with table from delta table but I want to do with delta file(.parq).
Ex in documentaiton whitch might work :
CREATE OR REFRESH STREAMING TABLE TTTT_cleaned
AS SELECT * FROM live.TTTT
This doesnt work
CREATE OR REFRESH STREAMING TABLE TTTT_cleaned
AS SELECT * FROM live.abfss://bronze-dev-sapboa@encrtpyion/somefolder/Delta/TTTT
how I can create from these files?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-08-2024 06:45 AM
Dear Kaniz,
Thank you for addressing question :
I am getting following error if i follow above:
pyspark.errors.exceptions.captured.IllegalArgumentException: Reading from a Delta table is not supported with this syntax. If you would like to consume data from Delta, please refer to the docs: read a Delta table (https://docs.microsoft.com/azure/databricks/delta/tutorial#read), or read a Delta table as a stream source (https://docs.microsoft.com/azure/databricks/structured-streaming/delta-lake#table-streaming-reads-an... The streaming source from Delta is already optimized for incremental consumption of data.
-----------------------------------------
Now my question is, if at all works... is it good practice to keep steam table on bronze these incremental (delta) parq files which are in my bronze layer.... ?
