Wrong FS: abfss://....., expected: dbfs:/ Error in DLT pipeline

harvey-c — Tue, 19 Dec 2023 02:42:46 GMT

Dear Databricks community members:

Symptom

Received the error for a delta load, after a successful initial load with a Unity Catalog Volume as a data source.

org.apache.spark.sql.streaming.StreamingQueryException: [STREAM_FAILED] Query [id = xxx, runId = xxx] terminated with exception: Wrong FS: abfss://container@storageaccount.dfs.core.windows.net/data.csv, expected: dbfs:/

Context

The Azure storage source was defined as a volume with external location. The DLT table was defined with dlt.readStream. and executed successfully. I changed the load path to next month in the same container, then the following load was failed with the above error.

Could you please advise what might go wrong and how to fix the issue?

Thank you!

topic Wrong FS: abfss://....., expected: dbfs:/ Error in DLT pipeline in Data Engineering

Wrong FS: abfss://....., expected: dbfs:/ Error in DLT pipeline