Data Lakehouse architecture with Azure Databricks and Unity Catalog

Pratikmsbsvm — Wed, 25 Jun 2025 16:08:02 GMT

I am Creating a Data lakehouse solution on Azure Databricks.

Source : SAP, SALESFORCE, Adobe

Target: Hightouch (External Application), Mad Mobile (External Application)

The datalake house also have transactional records which should be store in ACID property storage.

The real challenge is there is 1 more Databricks instance which is on seperate instance.

and that also required data from Data lakehouse.

May someone please help me how architecture should looks like.

Thanks a lot.

Re: Data Lakehouse architecture with Azure Databricks and Unity Catalog

CURIOUS_DE — Wed, 25 Jun 2025 17:59:25 GMT

@Pratikmsbsvm
You can leverage the below one for your architect solution.

Your Setup at a Glance

Sources

SAP, Salesforce, Adobe (Structured & Semi-structured)

Targets

Hightouch, Mad Mobile (External downstream apps needing curated data)

Core Requirement

Data must be stored in ACID-compliant format → ✅ Use Delta Lake(Managed will be great/ if there are company constraint external location will work)

Cross-Workspace Data Sharing

Another Databricks instance (separate workspace) needs access to this lakehouse data

Recommended Architecture (High-Level View)

[ SAP / Salesforce / Adobe ]
│
▼
Ingestion Layer (via ADF / Synapse / Partner Connectors / REST API)
│
▼
┌───────────────────────────┐
│ Azure Data Lake Gen2 │ (Storage layer - centralized)
│ + Delta Lake for ACID │
└───────────────────────────┘
│
▼
Azure Databricks (Primary Workspace)
├─ Bronze: Raw Data
├─ Silver: Cleaned & Transformed
└─ Gold: Aggregated / Business Logic Applied
│
├──> Load to Hightouch / Mad Mobile (via REST APIs / Hightouch Sync)
└──> Share curated Delta Tables to Other Databricks Workspace (via Delta Sharing or External Table Mount)

Key Components & Patterns

1. Ingestion Options

Use Azure Data Factory or Partner Connectors (like Fivetran- We use it often our project) to ingest data from:
- SAP → via OData / RFC connectors
- Salesforce → via REST/Bulk API
- Adobe → via API or S3 data export

2. Storage & Processing Layer

Store all raw and processed data in ADLS Gen2, with Delta Lake format
Organize Lakehouse zones:
- Bronze: Raw ingested files
- Silver: Cleaned & de-duplicated
- Gold: Ready for consumption (BI / API sync)

5. Cross-Workspace Databricks Access (This is Your Core Challenge and most important)

Option A: Delta Sharing (Recommended if in different orgs/subscriptions)

Securely share Delta tables from one workspace to another without copying data
Works across different cloud accounts

Option B: Mount/Use Service Principal ADLS Storage Account (Only if workspaces are under same Azure AD tenant)

Mount/use Service Principal same ADLS Gen2 storage in both Databricks workspaces
Other workspace can directly access tables if permissions are aligned in Groups (Access via Databricks Account Console)

Option C: Data Replication with Jobs

Periodically replicate key Delta tables to the secondary Databricks instance using jobs or autoloader

Governance / Security Recommendations

Use Unity Catalog (if available) for fine-grained access control
Encrypt data at rest (ADLS) and in transit
Use service principals or managed identities for secure access between services

Summary Visual (Simplified)

 Sources →           Ingestion →    Delta Lakehouse →            Destinations
[SAP, SFDC, Adobe]   [ADF, APIs]    [Bronze, Silver, Gold]      [Hightouch, Mad Mobile, Other DBX]
                                      ▲
                                      │
                                  Cross-Workspace Access (Delta Sharing / Mounting / Jobs)

Let me know if this helps 🙂

Re: Data Lakehouse architecture with Azure Databricks and Unity Catalog

KaranamS — Wed, 25 Jun 2025 18:44:08 GMT

Hi @Pratikmsbsvm , from what I understand, you have a lakehouse on Azure databricks and would like to share this data with another databricks account or workspace. If Unity Catalog is enabled on your Azure databricks account, you can leverage Delta Sharing to securely share the data with other databricks accounts.

https://docs.databricks.com/aws/en/delta-sharing/

Feel free to post if this does not answer your question or you need any specific details regarding this solution

topic Re: Data Lakehouse architecture with Azure Databricks and Unity Catalog in Data Engineering

Data Lakehouse architecture with Azure Databricks and Unity Catalog

Re: Data Lakehouse architecture with Azure Databricks and Unity Catalog

Recommended Architecture (High-Level View)

Key Components & Patterns

1. Ingestion Options

2. Storage & Processing Layer

5. Cross-Workspace Databricks Access (This is Your Core Challenge and most important)

Option A: Delta Sharing (Recommended if in different orgs/subscriptions)

Option B: Mount/Use Service Principal ADLS Storage Account (Only if workspaces are under same Azure AD tenant)

Option C: Data Replication with Jobs

Governance / Security Recommendations

Summary Visual (Simplified)

Re: Data Lakehouse architecture with Azure Databricks and Unity Catalog