Delta Live Table Graph different with no change in Notebook code

Manjula_Ganesap
Contributor

‎09-06-2023 05:05 AM

I have a DLT code to create 40+ bronze tables. The tables are created on top of the latest parquet files for each of those tables. 

While executing the pipeline, sometimes I notice that the graph is different than the regular one i see. I do not understand why that happens. 

Graph i expect to see:

Manjula_Ganesap_0-1694001824330.png

Graph i see sometimes:

Manjula_Ganesap_1-1694001865082.png

As you can see from the screenshots - only 2 tables were created in the second run while all 40+ tables were created in the first run. The underlying parquet files were the same in both cases. 

0 Kudos

Manjula_Ganesap
Contributor

‎09-06-2023 08:27 AM

@Retired_mod  - Thank you for your response. There is no change in the table dependencies. 

The code to create the individual raw tables look like this: The input to this is always the same 40 tables with only the underlying parquet file changing. I cant understand why it creates 40 tables in the first run and then only 2 tables in the second run.

def CreateTable(tableSchema,tableName, tableFilePath):
  schemaTableName = 'test_dlt_'+tableName.lower()
  @dlt.table(
    name= schemaTableName,
    comment="Raw data capture for " + tableName,
    table_properties={
      "quality": "bronze",
      "pipelines.autoOptimize.managed": "true"
    }
   )
  def create_live_table():
    return (
     (spark.read.format("parquet").load(tableFilePath))
    )  
0 Kudos