I keep getting dataset from spark.table command (i...

Nastia · ‎06-18-2024

I am trying to create a simple dlt pipeline:

@dlt.table

def today_latest_execution():

return spark.sql("SELECT * FROM LIVE.last_execution")

@on_event_hook

def write_events_to_x(event😞

if (

today_latest_execution().count() == 0

😞

try:

...

And I am getting and error:

'Dataset' object has no attribute 'count'

What I have tried: convertion to pandas (via ToPandas() or to_pandas_on_spark doesn't work), koalas doesn't work, using diff functions (not spark.sql) doesn't work... I am stuck 😞

How to make my function return me dataframe instead of dataset?

I keep getting dataset from spark.table command (instead of dataframe)