topic Re: Data Modelling in Get Started Discussions

Data Modelling

sravs1 — Fri, 27 Dec 2024 12:06:44 GMT

What is the 'implicit' or 'by default' data model of databricks or unity catalog ? Is it Data Vault ?

Re: Data Modelling

Alberto_Umana — Fri, 27 Dec 2024 12:53:59 GMT

The implicit or default data model used by Databricks and Unity Catalog is not Data Vault. Instead, Unity Catalog employs a hierarchical object model to organize and manage data. This hierarchy consists of three levels:

Catalogs: The top-level containers used to organize data assets, often mirroring organizational units or software development lifecycle scopes.
Schemas: Also known as databases, these contain tables, views, volumes, AI models, and functions.
Tables and Views: Tables are collections of data organized by rows and columns, while views are saved queries against one or more tables.

Unity Catalog provides centralized access control, auditing, lineage, and data discovery capabilities across Databricks workspaces. It operates on the principle of least privilege, ensuring users have the minimum access needed to perform their tasks

https://docs.databricks.com/en/data-governance/unity-catalog/index.html

Re: Data Modelling

szymon_dybczak — Fri, 27 Dec 2024 13:01:03 GMT

Hi @sravs1 ,

Databricks and Unity Catalog are desiged to support various data modeling approaches,. So you can implement your solution using data vault, star schema etc. Unity catalog doesn't impose any particular data modeling approach.

Re: Data Modelling

fmadeiro — Sat, 28 Dec 2024 01:55:27 GMT

Databricks and Unity Catalog do not enforce a specific data model like Data Vault. The default is a Lakehouse architecture using Delta Lake, which supports flexible schemas, ACID transactions, and schema evolution.

Unity Catalog organizes data into metastores, catalogs, schemas, and tables, allowing implementation of various models like Data Vault, Star Schema, or 3NF, depending on the use case.

For large-scale, auditable systems, Data Vault is a good fit and can leverage Delta Lake features for incremental loads and schema enforcement.

Refer to the Databricks Lakehouse Documentation and Unity Catalog Overview for more details.

Re: Data Modelling

ZacharyHill — Fri, 03 Jan 2025 06:19:38 GMT

Thank you so much for the information. You made it easy for me.