topic Re: Mount point in unity catalog in Data Engineering

Mount point in unity catalog

adityarai316 — Fri, 19 Jul 2024 17:25:22 GMT

Hi Everyone,

In my existing notebooks we have used mount points url as /mnt/ and we have more than 200 notebooks where we have used the above url to fetch the data/file from the container. Now as we are upgrading to unity catalog these url will no longer be supporting and we have to use something which start with abfss://

here my concern is I can’t go and edit each and every notebooks with new url. Is there any way to achieve this by doing minimal changes at notebook level by keeping old url which should refer to new url in unity catalog.

Thanks in advance for your time and support!!

#unitycatalog

Re: Mount point in unity catalog

Witold — Fri, 26 Jul 2024 10:05:58 GMT

@adityarai316 Can you use Volumes instead? It basically replaces the old mount approach

Re: Mount point in unity catalog

adityarai316 — Thu, 01 Aug 2024 11:02:09 GMT

Fine but for stg and prod url will be different right if i replace stg url in notebook and if i move it into prod then stg url will not work in prod right

Re: Mount point in unity catalog

adityarai316 — Thu, 01 Aug 2024 11:03:53 GMT

But in existing notebooks it is like /mnt/ so any way we have to replace this url with something else right so that it can point to new url. Volume will basically point to abfss url right ?

Re: Mount point in unity catalog

Witold — Thu, 01 Aug 2024 12:38:08 GMT

@adityarai316 Yes, they also point to abfss urls as well. The main and very important difference though is that volumes are "database objects", and with unity catalog in place you can put access control on that, i.e. granting read/write access to different groups/users. This is not possible to "old-school" mounts.

Re: Mount point in unity catalog

NaveenBedadala — Mon, 28 Oct 2024 05:09:15 GMT

@adityarai316 did u get the solution because I am facing the same issue?

Re: Mount point in unity catalog

adityarai316 — Mon, 20 Jan 2025 02:39:47 GMT

yes, we have stored old mount point and new url in form of key-value pair at the cluster level and defined it as a global variable. Post that we have used it in our custom notebook by importing os cmd and the variable. This way you will be able to tackle it.