Troubleshooting AutoLoader

[ChristianRRL]

Hi there, I am running into a bit of an issue displaying some AutoLoader readStream data. Can I get some assistance to understand how to properly troubleshoot this? I've looked at logs before, but frankly it's not clear where to look exactly:

First, "No rows returned" comes up after a few seconds (5-10 seconds)

Then, this error message comes up shortly after: "Error while trying to fetch latest data. Please check Driver logs."

[MartinIsti]

I'm also working with AutoLoader these days to create an ingestion pattern and troubleshooting it can be tricky.

I wonder if you could pick a single file (whose full path / location / uri you exactly know) and read it without autoloader. Just with spark.read.json ... or SELECT * FROM JSON.`path`

If that fails then it's not an autoloader issue. If that works, that might help further troubleshooting. Sometimes the order of the .option blocks can have an impact (it shouldn't ...), I experienced that just yesterday so keep an eye out for that, too.