topic Databricks recommended Approach to load data vault 2.0 in Data Engineering

Databricks recommended Approach to load data vault 2.0

Subha0920 — Tue, 05 Aug 2025 06:57:21 GMT

Hi,

Please share the recommended approach to load Data Vault 2.0 .

Overview

1. Current Landscape - Lakehouse (Bronze/Silver/Gold)

2. Data Vault 2.0 to be created in Silver layer.

3. Bronze data will be made available in delta table using ETL

Questions

1. What should be the strategy to load the data from Bronze to Silver layer

2. Approach to adopt to parallelize the load while loading the data vault 2.0 tables.

3.How to pick the incremental the data from delta tables while loading Silver layer.

4a)How can we reuse the Notebooks to load the Silver layer (Data Vault 2.0) for other source system.

b)Where should the logic to be encapsulated while populating hub/link/satellite table for every entity . ex views

c)How to configure the DQ Rules for every entity / tables

5. What type of meta data driven approach can be adopted.

6. What should be convention to adopt for Unity Catalog

ex - Unity Catalog Name - Bronze , Schema Name- Source System Name, Tables - Tables for every source.

Unity Catalog Name - Silver , Schema - what need to be provided . Tables - Data Vault 2.0 tables.

7. Exception Handling / Reprocessing from the point of failure / Auditing

8. Cluster Configuration (All purpose Cluster ) / Warehouse Cluster

Re: Databricks recommended Approach to load data vault 2.0

ilir_nuredini — Tue, 05 Aug 2025 21:06:53 GMT

Hello @Subha0920 ,

I have implemented previously data vault 2.0 in Databricks, even though it can be too long to mention all the details
of the implementation, but what helped us to get a lot of insights are these resource by Microsoft:
Data Vault 2.0 using Databricks Lakehouse Architecture on Azure

What’s a Data Vault and How to Implement It on the Databricks Lakehouse Platform - The Databricks Blog

They may be a bit old articles, but they are quite a helpful ones.

Hope that helps a bit.

Best, Ilir

Re: Databricks recommended Approach to load data vault 2.0

Subha0920 — Wed, 06 Aug 2025 04:24:30 GMT

Thanks @ilir_nuredini . It is helpful.

If you can share the details for the above questions, that will assist to plan further.

Re: Databricks recommended Approach to load data vault 2.0

Subha0920 — Thu, 07 Aug 2025 10:21:07 GMT

Kindly provide your valuable input and suggestion for the above questions