Where exactly I should create Volume in a catalog?

[hpant]

Currently my Databricks looks like this: 

I want to create volume to access external location. Where exactly should I create it? Should a create new schema in "poe" catalog and create a volume inside it or create it in a existing schema? What is the best practice?

[holly · Accepted solution]

Hello! Volumes go inside of schemas (screenshot below). It's up to you how to keep your data organised, but a few considerations:

• If you're going to have lots of volumes, does it make sense to group them together?
• As it's raw data, it's probably categorised as 'bronze' data, you could consider keeping it with that  
• Will you have to manage access to this data? Does it make sense to group it with other data you may want to prevent / promote access to?
• Do you want it to inherit other properties from things in the schema like tags, features or access patterns?

At the end of the day, schemas and catalogs are there to keep data organised. With external volumes (and tables) it has no baring on where the data is stored so it doesn't have much technical impact. 

My messy demo example:

[hpant]

Alright, thanks for your explanation. I have one more question, after creating a volume, how would you get it connect to a container? Imagine, you have created volume at external location a and you want to connect it external location b?

[holly]

Hi hpant, each volume is mapped to one location only. If you need to get data from two different locations, you'd make two separate volumes and join them as part of your pipeline.

If you wanted to read in from one location and write to another, again, you'd do that with two separate volumes. 

When I said 'group them together' above - you can have multiple volumes in one schema, even if their locations are very different. 

[hpant]

Hey, thanks for your response. currently, I have my data in one of the container in azure which gets added to the container through azure factory pipeline.. I have created a unity catalog workspace in different resource group. It has a container but there is no data in it. I have created a volume in it. Now I want to connect the volume to  the data present in a container of different storage account of different resource group. How can I make that connection? Do I need some sort of access key mechanism?

[Kaniz_Fatma]

Hi @hpant, Thanks for reaching out! Please review the responses and let us know which one best addresses your question. Your feedback is valuable to us and the community.

If the response resolves your issue, kindly mark it as the accepted solution. This will help close the thread and assist others with similar queries.

We appreciate your participation and are here if you need further assistance!

[holly]

Hi hpant,

You need to set up a new volume using a new external location (and potentially storage credential). Docs here: https://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/sql-ref-external-locations

[hpant1]

Hi @holly ,

Thanks so much for your response. I have one last question in this regard. Whenever I want add an extra location, ( external location), do I need to give Contributor role or higher on the access connector resource in Azure to add the storage credential first?

Create a storage credential for connecting to Azure Data Lake Storage Gen2 - Azure Databricks | Micr...

Thanks,

Hiamnshu Pant

[holly]

The docs say 'Contributor or Owner of an Azure resource group' and I don't have any reason to contradict that  

[Kaniz_Fatma]

Hi @hpant, @hpant1, If the response resolves your issue, kindly mark it as the accepted solution. This will help close the thread and assist others with similar queries.

[hpant1]

@Kaniz_Fatma .

I am trying to do that but couldn't find the option.

[Kaniz_Fatma]

Hi @hpant1, Do you have a similar view?

[hpant1]

No, I don't have. 

[Kaniz_Fatma]

Hi @hpant1, Okay, for now, I'll accept the solution on your behalf. I'll also look into why you're not seeing the option to accept a solution under your profile.