Wrong FS: abfss://....., expected: dbfs:/ Error in DLT pipeline

[harvey-c]

Dear Databricks community members:

Symptom

Received the error for a delta load, after a successful initial load with a  Unity Catalog Volume as a data source.

org.apache.spark.sql.streaming.StreamingQueryException: [STREAM_FAILED] Query [id = xxx, runId = xxx] terminated with exception: Wrong FS: abfss://container@storageaccount.dfs.core.windows.net/data.csv, expected: dbfs:/

Context

The Azure storage source was defined as a volume with external location. The DLT table was defined with dlt.readStream. and executed successfully. I changed the load path to next month in the same container, then the following load was failed with the above error.

Could you please advise what might go wrong and how to fix the issue? 

Thank you!

[Kaniz]

Hi @harvey-c, The file system path of the data source has changed from dbfs:/ to abfss:// after a previous successful load. This might confuse the Spark streaming query and cause it to fail with a wrong file system exception.

One possible solution is to use the dbfsUri option when creating the streaming query and specify the correct file system path of the data source. For example, if your data source is in Azure blob storage, you can use something like this:

df = spark.readStream.format("csv").option("dbfsUri", "dbfs:/container@storageaccount.dfs.core.windows.net/data.csv").load()

Alternatively, you can also use the spark.readStream.format("csv").option("path", "abfss://container....

I hope this helps you resolve your issue. If you have any other questions, please feel free to ask. 😊