Moving files from ADLS to Unity Catalog External Volume
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-02-2024 03:12 AM
Hi All,
I am trying to migrate files from ADLS to newly created UC External Volume.
I tried using dbutils.fs.cp but failed.
May I know what is the right way to copy files without any transformation from ADLS to UC external volume.
Thank you
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-02-2024 03:24 AM
You don't need to move / copy data at all. Just link the new volume to the existing storage location.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-02-2024 03:33 AM
The problem here is, the old ADLS storage will be dismantled after we move to Unity Catalog.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-02-2024 03:41 AM - edited 09-02-2024 03:41 AM
Solve this in Azure, by using e.g. az copy / Storage Explorer to move the data to new location. The volume should use this new location.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-02-2024 03:37 AM
Instead of copying your files from ADLS to UC volumes, you can create a storage credential and an external location. This allows you to access all your ADLS data directly through the catalog explorer under external locations. For guidance on creating a storage credential and external location, check out this YouTube video: https://youtu.be/KBB14qcELD4?si=uHcX2UQJXk-o2aJR

