Explicit Schema Definition: When calling spark.createDataFrame(pdf_cleaned), explicitly provide the schema even if the DataFrame is empty. This helps Spark infer the types and prevents the “cannot infer schema from empty dataset” error.
Guard Against Empty DataFrames: Check if pdf_cleaned is empty before creating a Spark DataFrame. If it’s empty, create a dummy DataFrame (with the right schema) instead
I agree with @szymon_dybczak and @BS_THE_ANALYST There isn’t a safe “hack” to force DLT dependency order when mixing Spark and Pandas APIs inside declarative tables, because DLT (and Lakeflow Pipelines) relies on dependency inference based on dlt.read() calls and doesn’t always guarantee materialization or downstream table population before execution, particularly when converting to/from Pandas
Knowledge base article calling this limitation - https://kb.databricks.com/delta-live-tables
