Troubleshooting AutoLoader

ChristianRRL
Honored Contributor

‎08-06-2025 08:56 AM

Hi there, I am running into a bit of an issue displaying some AutoLoader readStream data. Can I get some assistance to understand how to properly troubleshoot this? I've looked at logs before, but frankly it's not clear where to look exactly:

First, "No rows returned" comes up after a few seconds (5-10 seconds)

ChristianRRL_2-1754495673026.png

Then, this error message comes up shortly after: "Error while trying to fetch latest data. Please check Driver logs."

ChristianRRL_1-1754495653701.png

0 Kudos

MartinIsti
Databricks Partner

‎08-06-2025 03:30 PM

I'm also working with AutoLoader these days to create an ingestion pattern and troubleshooting it can be tricky.

I wonder if you could pick a single file (whose full path / location / uri you exactly know) and read it without autoloader. Just with spark.read.json ... or SELECT * FROM JSON.`path`

If that fails then it's not an autoloader issue. If that works, that might help further troubleshooting. Sometimes the order of the .option blocks can have an impact (it shouldn't ...), I experienced that just yesterday so keep an eye out for that, too.

0 Kudos