cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Delta Live Table Graph different with no change in Notebook code

Manjula_Ganesap
Contributor

I have a DLT code to create 40+ bronze tables. The tables are created on top of the latest parquet files for each of those tables. 

While executing the pipeline, sometimes I notice that the graph is different than the regular one i see. I do not understand why that happens. 

Graph i expect to see:

Manjula_Ganesap_0-1694001824330.png

Graph i see sometimes:

Manjula_Ganesap_1-1694001865082.png

As you can see from the screenshots - only 2 tables were created in the second run while all 40+ tables were created in the first run. The underlying parquet files were the same in both cases. 

1 REPLY 1

Manjula_Ganesap
Contributor

@Retired_mod  - Thank you for your response. There is no change in the table dependencies. 

The code to create the individual raw tables look like this: The input to this is always the same 40 tables with only the underlying parquet file changing. I cant understand why it creates 40 tables in the first run and then only 2 tables in the second run.

def CreateTable(tableSchema,tableName, tableFilePath):
  schemaTableName = 'test_dlt_'+tableName.lower()
  @dlt.table(
    name= schemaTableName,
    comment="Raw data capture for " + tableName,
    table_properties={
      "quality": "bronze",
      "pipelines.autoOptimize.managed": "true"
    }
   )
  def create_live_table():
    return (
     (spark.read.format("parquet").load(tableFilePath))
    )  

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group