topic Re: Autoloader Error Loading and Displaying in Data Engineering

Autoloader Error Loading and Displaying

ChristianRRL — Mon, 23 Jun 2025 18:30:16 GMT

Hi there,

I'd appreciate some assistance with troubleshooting what is supposed to be a (somewhat) simple use of autoloader. Below are some screenshots highlighting my issue:

When I attempt to create the dataframe via spark.readStream.format("cloudFiles"), a dataframe with the correct nested structure seems to be created, but when I attempt to run display on the dataframe, I get the following error message:

Error while trying to fetch latest data. Please check Driver logs.

I've tried checking the logs, but to be honest they're not very clear.

Re: Autoloader Error Loading and Displaying

lingareddy_Alva — Mon, 23 Jun 2025 20:22:35 GMT

Hi @ChristianRRL

This is a common issue with Spark Structured Streaming and the display() function.
The error occurs because you're trying to display a streaming DataFrame, which requires special handling. Here are several solutions:

1. Use writeStream instead of display()
For streaming DataFrames, use writeStream to output the data:

# Instead of display(df)
query = (df.writeStream
.format("console") # or "memory", "delta", etc.
.outputMode("append") # or "complete", "update"
.trigger(once=True) # Process once then stop
.start())

query.awaitTermination()

2. Use Memory Sink for Testing:
Create a temporary view to examine streaming data:
# Start the stream writing to memory
query = (df.writeStream
.format("memory")
.queryName("temp_table")
.outputMode("append")
.start())

# Wait a moment for data to be processed
import time
time.sleep(10)

# Now you can query the in-memory table
display(spark.sql("SELECT * FROM temp_table LIMIT 10"))

# Don't forget to stop the query
query.stop()

The key issue is that display() doesn't work with streaming DataFrames - you need to use writeStream to materialize the data first.

Re: Autoloader Error Loading and Displaying

ChristianRRL — Mon, 04 Aug 2025 22:02:50 GMT

Hey @lingareddy_Alva , thanks a ton for the input. I went ahead and tried both methods you suggested. I am seeing better luck with the 2nd method, although I personally wouldn't want to add a 10 second wait while developing/debugging. Can I get help to understand why the 1st method isn't working?