https://community.databricks.com/t5/data-engineering/custom-application-log-files-copy-fails-in-shared-unity-catalog/m-p/4177#M975 <P>We have a use case of implementing custom logging framework for capturing the various details while executing a notebook. At the end of the notebook, the log file gets copied from driver's /tmp folder into Data lake Storage's container.</P><P></P><P>While this process works perfectly in normal all purpose clusters, it has been failing ever since we moved on to Shared Unity Catalog cluster.</P><P></P><P>We use this below code to copy the file into ABFSS path. A separate external location has also been defined for the purpose of log files.</P><P></P><P>dbutils.fs.cp("file:"+log_file_path, "abfss://application-logs@&lt;storageaccount&gt;.dfs.core.windows.net/adb" + date_folders + log_file_name, True)</P> Thu, 18 May 2023 21:36:54 GMT rk_db 2023-05-18T21:36:54Z https://community.databricks.com/t5/data-engineering/custom-application-log-files-copy-fails-in-shared-unity-catalog/m-p/4177#M975 <P>We have a use case of implementing custom logging framework for capturing the various details while executing a notebook. At the end of the notebook, the log file gets copied from driver's /tmp folder into Data lake Storage's container.</P><P></P><P>While this process works perfectly in normal all purpose clusters, it has been failing ever since we moved on to Shared Unity Catalog cluster.</P><P></P><P>We use this below code to copy the file into ABFSS path. A separate external location has also been defined for the purpose of log files.</P><P></P><P>dbutils.fs.cp("file:"+log_file_path, "abfss://application-logs@&lt;storageaccount&gt;.dfs.core.windows.net/adb" + date_folders + log_file_name, True)</P> Thu, 18 May 2023 21:36:54 GMT https://community.databricks.com/t5/data-engineering/custom-application-log-files-copy-fails-in-shared-unity-catalog/m-p/4177#M975 rk_db 2023-05-18T21:36:54Z