Hello,
I am not sure if this is the right place to post this, however, I reporting what seems to me, a breaking issue with 16.3 Beta Runtime, when performing dbutils.fs.cp() operations between abfss://
This is not a permissions issue — let's get that out of the way. It's something that has been working for years, and switching to the 16.2 Runtime works — everything equal.
Let's assume:
SOURCE_URI = 'abfss://...'
DESTINATION_URI = 'abfss://...'
dbutils.fs.cp(SOURCE_URI, DESTINATION_URI, True)
Throws exception:
ExecutionError: An error occurred while calling o595.cp.
: java.lang.ClassCastException: class com.databricks.common.filesystem.LokiAbfsInputStream cannot be cast to class shaded.databricks.azurebfs.org.apache.hadoop.fs.azurebfs.services.AbfsInputStream (com.databricks.common.filesystem.LokiAbfsInputStream and shaded.databricks.azurebfs.org.apache.hadoop.fs.azurebfs.services.AbfsInputStream are in unnamed module of loader 'app')
at com.databricks.common.filesystem.LokiABFS.setModTimeSettings(LokiABFS.scala:83)
at com.databricks.common.filesystem.LokiABFS.openFileWithOptions(LokiABFS.scala:65)
at org.apache.hadoop.fs.FileSystem$FSDataInputStreamBuilder.build(FileSystem.java:4913)
at com.databricks.common.filesystem.LokiFileSystem.openFileWithOptions(LokiFileSystem.scala:316)
at org.apache.hadoop.fs.FileSystem$FSDataInputStreamBuilder.build(FileSystem.java:4913)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:488)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:430)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:368)
at com.databricks.backend.daemon.dbutils.FSUtils.cpRecursive(DBUtilsCore.scala:659)
at com.databricks.backend.daemon.dbutils.FSUtils.$anonfun$cpRecursive$1(DBUtilsCore.scala:637)
at com.databricks.backend.daemon.dbutils.FSUtils.$anonfun$cpRecursive$1$adapted(DBUtilsCore.scala:636)
at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198)
...
Seems to be related to this new 'app' concept, maybe.
Is this a known breaking change that we need to be aware of? Surely, I hope not.
