Hello @pernilak ,

Thanks for reaching out to Databricks Community! My name is Raphael, and I'll be helping out.

Should all catalogs and the metastore reside in the same storage account (but different containers)

Yes, Databricks recommends having one separate storage location (container) per catalog. But you can also have one single container for the whole metastore (metastore-level storage). If you need to isolate your data at infrastructure level (i.e separate storage accounts) then the best practice is to use External Locations but you can't create a whole catalog in an external location, only other small entities such as tables or volumes.

For information to help you decide whether you need metastore-level storage, see (Optional) Create metastore-level storage and Data is physically separated in storage.

Should the metastore have one storage account and other catalogs reside in a different one (separate containers)

Metastore and catalogs should reside in the same storage account, you can have one container per catalog or one container for all metastore entities, it's up to you to decide. My answer for your question no.1 has the auxiliar doc urls that should help you understand which option is better to you.

Should dev, test and prod catalogs be in different storage accounts?

I don't think that it's possible, if you want to work with separate storage accounts then you should use External Locations.

Should one domain (we have catalogs based on domain) be in one storage account, but then have dev, test and prod catalogs in different containers?

This is a good pattern.

Should data be separated based on the requirements for retention and backup?

Not necessarily, but it can be done. With UC, data retention and backup will mostly rely on your cloud storage retention policies/backup policies. Databricks itself allows for table-level short-term backups (https://docs.databricks.com/en/delta/history.html) while also always respecting the cloud storage policies.

Or should we separate data on schemas (different containers or storage accounts?)?

This is a good pattern but you must use a single storage account for storing these schemas. 

1. If a location has been provided for mySchema, it will be stored there.
2. If not, and a location has been provided on myCatalog, it will be stored there.
3. Finally, if no location has been provided on myCatalog, it will be stored in the location associated with the my-region-metastore.

Should some schemas not reside in the same storage account as the catalog?

I don't think that it's possible, but you can have external tables within another storage accounts stored under one of your catalog's schemas. You can also have external volumes (also in separate storage accounts) for storing/fetching files in your Unity Catalog.

Final observations:

Let's say you have a storage account no.1 and no.2. Then you choose no.1 to create your metastore and you create your dev catalog there.

But, you do have some tables in storage account no.2 that you'd like to use in your UC dev catalog stored in storage account no.1.

If this is the case, then you'll be creating an external table on your dev catalog pointing to storage account no.2. But what happens with your data and metadata?

Data -> Stored under storage account no.2

Metadata -> Stored under storage account no.1 (dev catalog in this example)

Feel free to ask any further questions, if my response addresses your concerns then please mark it as the official solution 🙂 

Thanks!