dbutls.fs.cp() fails in Runtime 16.3 Beta, when using abfss://

kleanthis — Wed, 26 Mar 2025 09:17:24 GMT

Hello,

I am not sure if this is the right place to post this, however, I reporting what seems to me, a breaking issue with 16.3 Beta Runtime, when performing dbutils.fs.cp() operations between abfss://

This is not a permissions issue — let's get that out of the way. It's something that has been working for years, and switching to the 16.2 Runtime works — everything equal.

Let's assume:

SOURCE_URI = 'abfss://...'

DESTINATION_URI = 'abfss://...'

dbutils.fs.cp(SOURCE_URI, DESTINATION_URI, True)

Throws exception:
ExecutionError: An error occurred while calling o595.cp.
: java.lang.ClassCastException: class com.databricks.common.filesystem.LokiAbfsInputStream cannot be cast to class shaded.databricks.azurebfs.org.apache.hadoop.fs.azurebfs.services.AbfsInputStream (com.databricks.common.filesystem.LokiAbfsInputStream and shaded.databricks.azurebfs.org.apache.hadoop.fs.azurebfs.services.AbfsInputStream are in unnamed module of loader 'app')
at com.databricks.common.filesystem.LokiABFS.setModTimeSettings(LokiABFS.scala:83)
at com.databricks.common.filesystem.LokiABFS.openFileWithOptions(LokiABFS.scala:65)
at org.apache.hadoop.fs.FileSystem$FSDataInputStreamBuilder.build(FileSystem.java:4913)
at com.databricks.common.filesystem.LokiFileSystem.openFileWithOptions(LokiFileSystem.scala:316)
at org.apache.hadoop.fs.FileSystem$FSDataInputStreamBuilder.build(FileSystem.java:4913)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:488)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:430)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:368)
at com.databricks.backend.daemon.dbutils.FSUtils.cpRecursive(DBUtilsCore.scala:659)
at com.databricks.backend.daemon.dbutils.FSUtils.$anonfun$cpRecursive$1(DBUtilsCore.scala:637)
at com.databricks.backend.daemon.dbutils.FSUtils.$anonfun$cpRecursive$1$adapted(DBUtilsCore.scala:636)
at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198)

...

Seems to be related to this new 'app' concept, maybe.

Is this a known breaking change that we need to be aware of? Surely, I hope not.

Re: dbutls.fs.cp() fails in Runtime 16.3 Beta, when using abfss://

kleanthis — Wed, 02 Apr 2025 08:38:27 GMT

To close off my loop: The ClassCastException has been resolved in 16.3

topic Re: dbutls.fs.cp() fails in Runtime 16.3 Beta, when using abfss:// in Data Engineering

dbutls.fs.cp() fails in Runtime 16.3 Beta, when using abfss://

Re: dbutls.fs.cp() fails in Runtime 16.3 Beta, when using abfss://