topic Re: Logging to an external location via UC volume in Data Engineering

Logging to an external location via UC volume

dollyb — Thu, 12 Dec 2024 14:11:40 GMT

The way I understand it, mount points are deprecated in UC. db.fs.mount() doesn't even seem to work in newer DB runtimes.

But what is the solution when Databricks features don't allow using UC volumes? E.g. specifying a compute's logging path won't work with volumes.

Re: Logging to an external location via UC volume

Walter_C — Thu, 12 Dec 2024 14:25:43 GMT

Right now only mount supported option is to use Volumes as you have mentioned, otherwise you can connect directly to the cloud storage object https://docs.databricks.com/en/connect/unity-catalog/index.html

Re: Logging to an external location via UC volume

dollyb — Thu, 12 Dec 2024 16:29:40 GMT

> otherwise you can connect directly to the cloud storage object

You mean using the cloud provider's APIs? That wouldn't solve logging to cloud storage as the the compute's logging path only allows dbfs:/ references. So can I only log to a managed DBFS location?

Re: Logging to an external location via UC volume

Walter_C — Thu, 12 Dec 2024 16:47:59 GMT

As you cannot use volumes it seems that indeed this will be your only option

Re: Logging to an external location via UC volume

dollyb — Fri, 13 Dec 2024 10:37:51 GMT

I'm just wondering why Notebook do support mounts, even in my UC workspace?

Re: Logging to an external location via UC volume

dollyb — Fri, 13 Dec 2024 12:22:23 GMT

I can actually see the mount point does exist when I do

DBUtils.getDBUtils().fs.mounts()

MountInfo(/mnt/cluster-logs,wasbs://xxx@xxx.blob.core.windows.net/cluster-logs/,)

But still no logs arrive.

Re: Logging to an external location via UC volume

dollyb — Fri, 13 Dec 2024 12:49:39 GMT

Found the error message at last: DatabricksServiceException: IO_ERROR: java.io.IOException: This request is not authorized to perform this operation using this permission. Please see the cause for further information.