How to build Data Pipeline to consume data from Adobe Campaign to Azure Databricks

[Datalight]

May Techie please help me design the pipeline with Databricks.

I don't have any control over Adobe.

How to set up a data pipeline that moves csv files from Adobe to ADLS Gen2 via a cron job, using Databricks.

where this cron job will execute ? how ADLS Gen2 can detect new files.

How is the data being transferred from Adobe to ADLS Gen2 using Databricks only.

How to handle New File arrival? 

I am trying to have Event Driven Architecture.

Limitation : The Workspace is not have Unity Catalog Enabled.

All the resources running on private subnets. No Public IP- NPIP = Yes

[Khaja_Zaffer · Accepted solution]

Hello @Datalight 

Good day!

Can I please know what do you mean by "you dont have any control over adobe"? 

I found similar case study over here: https://learn.microsoft.com/en-us/answers/questions/5533633/data-pipeline-to-push-files-from-externa...