Custom application log files copy fails in Shared Unity Catalog Cluster

[rk_db]

We have a use case of implementing custom logging framework for capturing the various details while executing a notebook. At the end of the notebook, the log file gets copied from driver's /tmp folder into Data lake Storage's container.

While this process works perfectly in normal all purpose clusters, it has been failing ever since we moved on to Shared Unity Catalog cluster.

We use this below code to copy the file into ABFSS path. A separate external location has also been defined for the purpose of log files.

dbutils.fs.cp("file:"+log_file_path, "abfss://application-logs@<storageaccount>.dfs.core.windows.net/adb" + date_folders + log_file_name, True)