cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Unity Catalog v.2.0?

DB1To3
Contributor

Are there plans for a v.2.0 of Unity Catalog?  I find the organization of tables in databricks to be very constrained and inflexible.  Given that UC tables have their origins in data lakes, you would think they would have brought a lot more flexibile names into UC than what we are forced to use today.  The rigid three-level names, in lowercase letters, are just terrible IMO.  (Especially when data is arriving into UC from other technologies that have much more flexibile names than what we have in this UC.)

Try googling "how to name databricks catalogs and schemas", and you will find too MANY strategies that are being invented as a workaround, to overcome arbitrary limitations enforced in the databricks environment.  They often involve concatenating unrelated concepts together with underscores (<environment>, <medallion>, <business_domain>, <business_unit>, <whatever>, etc).  

The most challenging limitations are the restrictive three-level names, and being forced to use lowercase letters at the table level.  These limitations are unnecessary.  In a large UC environment, names should always be hierarchical, especially if the underlying cloud storage fully supports HNS. 

Assuming there are certain legacy internals, within databricks, that cannot properly deal with hierarchical names, then why not just support aliases, defined during bootstrapping?

eg....
USING ALIAS mycatalog.presentation.sales = prod_catalog.GoldPresentation.NorthAmerica.Erp01.ExecutiveSummary.OutsideSales

SELECT * FROM mycatalog.presentation.sales


I think aliases are very important.  It is silly for end end users and reporting analysts to constantly sprinkle terms like "production", or "gold" all over their queries and notebooks, when they never use anything different (and some probably don't even have access to anything different.) 

0 REPLIES 0