topic Re: How to create Delta live tables in Silver layer in Get Started Discussions

How to create Delta live tables in Silver layer

Madalian — Fri, 08 Mar 2024 10:35:27 GMT

How to create Delta live tables in Silver layer

Hi DB Experts,

Having basic questions :

I am working on Madalian Architecture (B, S, G) Layers.

on B i am getting Delta files (Parq) format. with log folders. One folder for one table, multiple files are getting placed after certain size... But I dont see any Detla tables.

Now from these files I want to create straming table in "Silver" Layer... I am unable to find solution. Following example shows to create with table from delta table but I want to do with delta file(.parq).
Ex in documentaiton whitch might work :
CREATE OR REFRESH STREAMING TABLE TTTT_cleaned
AS SELECT * FROM live.TTTT
This doesnt work
CREATE OR REFRESH STREAMING TABLE TTTT_cleaned
AS SELECT * FROM live.abfss://bronze-dev-sapboa@encrtpyion/somefolder/Delta/TTTT
how I can create from these files?

Re: How to create Delta live tables in Silver layer

Madalian — Fri, 08 Mar 2024 14:45:39 GMT

Dear Kaniz,

Thank you for addressing question :

I am getting following error if i follow above:

pyspark.errors.exceptions.captured.IllegalArgumentException: Reading from a Delta table is not supported with this syntax. If you would like to consume data from Delta, please refer to the docs: read a Delta table (https://docs.microsoft.com/azure/databricks/delta/tutorial#read), or read a Delta table as a stream source (https://docs.microsoft.com/azure/databricks/structured-streaming/delta-lake#table-streaming-reads-and-writes). The streaming source from Delta is already optimized for incremental consumption of data.

-----------------------------------------

Now my question is, if at all works... is it good practice to keep steam table on bronze these incremental (delta) parq files which are in my bronze layer.... ?