Data Modelling

[sravs1]

What is the 'implicit' or 'by default' data model of databricks or unity catalog ? Is it Data Vault ?

[Alberto_Umana]

The implicit or default data model used by Databricks and Unity Catalog is not Data Vault. Instead, Unity Catalog employs a hierarchical object model to organize and manage data. This hierarchy consists of three levels:

1. Catalogs: The top-level containers used to organize data assets, often mirroring organizational units or software development lifecycle scopes.
2. Schemas: Also known as databases, these contain tables, views, volumes, AI models, and functions.
3. Tables and Views: Tables are collections of data organized by rows and columns, while views are saved queries against one or more tables.

Unity Catalog provides centralized access control, auditing, lineage, and data discovery capabilities across Databricks workspaces. It operates on the principle of least privilege, ensuring users have the minimum access needed to perform their tasks

https://docs.databricks.com/en/data-governance/unity-catalog/index.html

[szymon_dybczak]

Hi @sravs1 ,

Databricks and Unity Catalog are desiged to support various data modeling approaches,. So you can implement your solution using data vault, star schema etc. Unity catalog doesn't impose any particular data modeling approach.

[fmadeiro]

Databricks and Unity Catalog do not enforce a specific data model like Data Vault. The default is a Lakehouse architecture using Delta Lake, which supports flexible schemas, ACID transactions, and schema evolution.

Unity Catalog organizes data into metastores, catalogs, schemas, and tables, allowing implementation of various models like Data Vault, Star Schema, or 3NF, depending on the use case.

For large-scale, auditable systems, Data Vault is a good fit and can leverage Delta Lake features for incremental loads and schema enforcement.

Refer to the Databricks Lakehouse Documentation and Unity Catalog Overview for more details.