Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-12-2024 01:52 AM
I have taken the advice given by the documentation (However, you can include these functions outside of table or view function definitions because this code is run once during the graph initialization phase.) and moved the toPandas call to a function without a "@table" decorator. It is still referenced by it, so I don't know if that is the intention of the remark.
import dlt
def pivot_table(spark_df):
# removed the pivoting to simplify the example
pdf = spark_df.toPandas()
return spark.createDataFrame(pdf)
@dlt.table
def dlt_test_table_pivoted():
df = spark.read.table('test_table')
result_df = pivot_table(df)
return result_dfThis does work, so I am a step further now.
But if I try to add an intermediate table function I get the same error as before ("ValueError: can not infer schema from empty dataset")
import dlt
def pivot_table(spark_df):
pdf = spark_df.toPandas()
return spark.createDataFrame(pdf)
@dlt.table
def dlt_test_table():
return spark.read.table('test_table')
@dlt.table
def dlt_test_table_pivoted():
df = dlt.read('dlt_test_table')
result_df = pivot_table(df)
return result_df