Hi everyone,

I’m running into an issue with a Delta Live Tables (DLT) pipeline that processes a few transformation layers (raw → intermediate → primary → feature).

When I trigger the entire pipeline, it fails with the following error:
can not infer schema from empty dataset

The error happens at this line:

df_spark = spark.createDataFrame(df_cleaned) 

However, if I run the steps manually (table by table), everything works perfectly. Even more strangely, once I’ve run the layers manually, the full pipeline runs successfully afterward. This makes me think the issue is related to dependency resolution or execution timing in DLT.

Simplified example

Here’s a simplified version of my code:

import dlt
from pyspark.sql import functions as F

@dlt.table(name="bronze_table")
def bronze_table():
    return spark.read.table("source_table")

@dlt.table(name="silver_intermediate")
def silver_intermediate():
    df = dlt.read("bronze_table")
    return df.withColumn("processed_col", F.upper(F.col("some_col")))

@dlt.table(name="silver_primary")
def silver_primary():
    df = dlt.read("silver_intermediate")
    df = df.withColumn("year", F.substring(F.col("date_col"), 0, 4))
    pdf = df.pandas_api()
    pdf_filtered = pdf[pdf["year"].notnull()]
    return pdf_filtered.to_spark()

@dlt.table(name="silver_feature")
def silver_feature():
    df = dlt.read("silver_primary").pandas_api()
    pdf = df.to_pandas()
    pdf_cleaned = pdf.dropna()
    # This line fails when the pipeline runs end-to-end
    df_spark = spark.createDataFrame(pdf_cleaned)
    return df_spark

What I suspect

It seems that DLT might be running silver_feature before silver_primary has finished materializing, causing dlt.read("silver_primary") to return an empty dataset. When I run things manually, each dependency already exists, so it works fine.

Questions

1. Is there a known timing or dependency issue in DLT when chaining multiple transformations that mix Spark and Pandas API on Spark operations (and even pandas ops)?

2. Is there a way to ensure that DLT waits until an upstream table has data before running the next step?