Delta Live Table Graph different with no change in Notebook code

Manjula_Ganesap — Wed, 06 Sep 2023 12:05:51 GMT

I have a DLT code to create 40+ bronze tables. The tables are created on top of the latest parquet files for each of those tables.

While executing the pipeline, sometimes I notice that the graph is different than the regular one i see. I do not understand why that happens.

Graph i expect to see:

Graph i see sometimes:

As you can see from the screenshots - only 2 tables were created in the second run while all 40+ tables were created in the first run. The underlying parquet files were the same in both cases.

Re: Delta Live Table Graph different with no change in Notebook code

Manjula_Ganesap — Wed, 06 Sep 2023 15:27:26 GMT

@Retired_mod - Thank you for your response. There is no change in the table dependencies.

The code to create the individual raw tables look like this: The input to this is always the same 40 tables with only the underlying parquet file changing. I cant understand why it creates 40 tables in the first run and then only 2 tables in the second run.

def CreateTable(tableSchema,tableName, tableFilePath):

schemaTableName = 'test_dlt_'+tableName.lower()

@dlt.table(

name= schemaTableName,

comment="Raw data capture for " + tableName,

table_properties={

"quality": "bronze",

"pipelines.autoOptimize.managed": "true"

}

)

def create_live_table():

return (

(spark.read.format("parquet").load(tableFilePath))

)

topic Delta Live Table Graph different with no change in Notebook code in Data Engineering

Delta Live Table Graph different with no change in Notebook code

Re: Delta Live Table Graph different with no change in Notebook code