topic Re: Moving files from ADLS to Unity Catalog External Volume in Data Engineering

Moving files from ADLS to Unity Catalog External Volume

NC — Mon, 02 Sep 2024 10:12:07 GMT

Hi All,

I am trying to migrate files from ADLS to newly created UC External Volume.

I tried using dbutils.fs.cp but failed.

May I know what is the right way to copy files without any transformation from ADLS to UC external volume.

Thank you

Re: Moving files from ADLS to Unity Catalog External Volume

Witold — Mon, 02 Sep 2024 10:24:35 GMT

You don't need to move / copy data at all. Just link the new volume to the existing storage location.

Re: Moving files from ADLS to Unity Catalog External Volume

NC — Mon, 02 Sep 2024 10:33:35 GMT

The problem here is, the old ADLS storage will be dismantled after we move to Unity Catalog.

Re: Moving files from ADLS to Unity Catalog External Volume

navallyemul — Mon, 02 Sep 2024 10:37:31 GMT

Instead of copying your files from ADLS to UC volumes, you can create a storage credential and an external location. This allows you to access all your ADLS data directly through the catalog explorer under external locations. For guidance on creating a storage credential and external location, check out this YouTube video: https://youtu.be/KBB14qcELD4?si=uHcX2UQJXk-o2aJR

Re: Moving files from ADLS to Unity Catalog External Volume

Witold — Mon, 02 Sep 2024 10:41:45 GMT

Solve this in Azure, by using e.g. az copy / Storage Explorer to move the data to new location. The volume should use this new location.