bulbur
New Contributor II

I have taken the advice given by the documentation (However, you can include these functions outside of table or view function definitions because this code is run once during the graph initialization phase.) and moved the toPandas call to a function without a "@table" decorator. It is still referenced by it, so I don't know if that is the intention of the remark.

import dlt

def pivot_table(spark_df):
  # removed the pivoting to simplify the example
  pdf = spark_df.toPandas()
  return spark.createDataFrame(pdf)
 
@dlt.table
def dlt_test_table_pivoted():
  df = spark.read.table('test_table')
  result_df = pivot_table(df)
  return result_df

This does work, so I am a step further now.

But if I try to add an intermediate table function I get the same error as before ("ValueError: can not infer schema from empty dataset")

import dlt

def pivot_table(spark_df):
  pdf = spark_df.toPandas()
  return spark.createDataFrame(pdf)

@dlt.table
def dlt_test_table():
    return spark.read.table('test_table')
  
@dlt.table
def dlt_test_table_pivoted():
  df = dlt.read('dlt_test_table')
  result_df = pivot_table(df)
  return result_df