We are facing a similar issue, not limited to Autoloader but also affecting DLT pipelines and classic ETL job. The behavior is intermittent, jobs run fine and then fail unexpectedly, though they typically succeed on retry if retries are enabled. We tested with both absolute and relative paths, but the issue persists regardless.

I escalated this to our Databricks contact, and the suggested solutions are:

• Switch the channel from "Preview" to "Current" in the Databricks configuration, or
• Raise a support ticket with Microsoft or Databricks (depending on your hyperscaler), providing your workspace ID.

In our case, the issue appears to be specific to the West Europe region, so it seems Databricks is already aware of it and a fix is expected in an upcoming release.