Read Files from Adobe and Push to Delta table ADLS Gen2

[Pratikmsbsvm]

The Upstream is sending 2 files of different schema. 

The Storage Account has Private Endpoints. there is no public access.

no public IP (NPIP) = yes.

How to design using only Databricks :-

1. Databricks API to read data file from Adobe and Push it to ADLS Container.

2. Pulling new Data file whenever available. (Polling or pulling)

3. I want to replace Event Grid and Function App with Databricks , please help how to do that.

Thanks

[Khaja_Zaffer]

Hello @Pratikmsbsvm 

Good day

Here is the design for your requirements. 

Recommended Architecture (High-Level View)

[ SAP / Salesforce / Adobe ]
│
▼
Ingestion Layer (via ADF / Synapse / Partner Connectors / REST API)
│
▼
┌───────────────────────────┐
│ Azure Data Lake Gen2 │ (Storage layer - centralized)
│ + Delta Lake for ACID │
└───────────────────────────┘
│
▼
Azure Databricks (Primary Workspace)
├─ Bronze: Raw Data
├─ Silver: Cleaned & Transformed
└─ Gold: Aggregated / Business Logic Applied
│
├──> Load to Hightouch / Mad Mobile (via REST APIs / Hightouch Sync)
└──> Share curated Delta Tables to Other Databricks Workspace (via Delta Sharing or External Table Mount)

Key Components & Patterns

1. Ingestion Options

• Use Azure Data Factory or Partner Connectors (like Fivetran- We use it often our project) to ingest data from:

  • SAP → via OData / RFC connectors

  • Salesforce → via REST/Bulk API

  • Adobe → via API or S3 data export

2. Storage & Processing Layer

• Store all raw and processed data in ADLS Gen2, with Delta Lake format

• Organize Lakehouse zones:

  • Bronze: Raw ingested files

  • Silver: Cleaned & de-duplicated

  • Gold: Ready for consumption (BI / API sync)

Cross-Workspace Databricks Access (This is Your Core Challenge and most important)

Delta Sharing (Recommended if in different orgs/subscriptions)

• Securely share Delta tables from one workspace to another without copying data

• Works across different cloud accounts

Governance / Security Recommendations

• Use Unity Catalog (if available) for fine-grained access control

• Encrypt data at rest (ADLS) and in transit

• Use service principals or managed identities for secure access between services

Summary Visual (Simplified)

 Sources →           Ingestion →    Delta Lakehouse →            Destinations
[SAP, SFDC, Adobe]   [ADF, APIs]    [Bronze, Silver, Gold]      [Hightouch, Mad Mobile, Other DBX]
                                      ▲
                                      │
                                  Cross-Workspace Access (Delta Sharing / Mounting / Jobs)

Let me know if this helps 

Do you have any idea about APIs the code with you to connect from adobe to databricks?

[szymon_dybczak · Accepted solution]

Hi @Pratikmsbsvm ,

Okay, since you’re going to use Databricks compute for data extraction and you wrote that your workspace is deployed with the secure connectivity cluster (NPIP) option enabled, you first need to make sure that you have a stable egress IP address.

Assuming that your workspace uses VNET injection (and not a managed VNET), to add explicit outbound methods for your workspace, use an Azure NAT gateway or user-defined routes (UDRs):

• Azure NAT gateway: Use an Azure NAT gateway to provide outbound internet connectivity for your deployments with a stable egress public IP. Configure the gateway on both of the workspace's subnets to ensure that all outbound traffic to the Azure backbone and public network transits through it. Clusters have a stable egress public IP, and you can modify the configuration for custom egress needs. You can configure this using either an Azure template or from the Azure portal.

• UDRs: Use UDRs if your deployments require complex routing requirements or your workspaces use VNet injection with an egress firewall. UDRs ensure that network traffic is routed correctly for your workspace, either directly to the required endpoints or through an egress firewall. To use UDRs, you must add direct routes or allowed firewall rules for the Azure Databricks secure cluster connectivity relay and other required endpoints listed at User-defined route settings for Azure Databricks.

Once you have the stable egress IP issue sorted out, you will then need to write code to fetch the data from Adobe and save it to ADLS.
If your source data is in one of the following formats, I recommend using Auto Loader:

• avro : Avro files

• binaryFile : Binary files

• csv : CSV files

• json : JSON files

• orc : ORC files

• parquet : Parquet files

• text : TXT files

• xml : XML files

Auto Loader incrementally and efficiently processes new data files as they arrive in cloud storage. It provides a Structured Streaming source called cloudFiles. So to keep it simple, it will automatically detect that new files arrived on data lake and process only new files (with exactly once semantic).

You can connect Auto Loader with a file arrival trigger. So when new files arrive in the storage, an event will be generated that automatically starts the workflow to process the new files using autloader mechanism described above.

Trigger jobs when new files arrive - Azure Databricks | Microsoft Learn