Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-07-2024 01:28 AM
When setting up Unity Catalog, it is recommended by Databricks to figure out your data isolation model when it comes to physically separating your data into different storage accounts and/or contaners. There are so many options, it can be hard to be confident in the solution you choose. Some alternatives we are looking into are:
- Should all catalogs and the metastore reside in the same storage account (but different containers)
- Should the metastore have one storage account and other catalogs reside in a different one (separate containers)
- Should dev, test and prod catalogs be in different storage accounts?
- Should one domain (we have catalogs based on domain) be in one storage account, but then have dev, test and prod catalogs in different containers?
- Should data be separated based on the requirements for retention and backup?
- Or should we separate data on schemas (different containers or storage accounts?)?
- Should some schemas not reside in the same storage account as the catalog?
What are your thoughts on this subject. What are the pros and cons of the different methods based on your experience?