Adobe Campaign to Azure Databricks file transfer

Datalight
Contributor

I have to create a Data Pipeline which pushes Data (2. JSON FILE)  from Source Adobe using corn job to ADLS Gen2.

Datalight_0-1755698274906.png

 

1. How My ADLS Gen2 will know the new file came to container from Adobe. I am using Databricks as orchestrator and ETL tool.

2. What all are my network and Security approach when data moved from Adobe to ADLS Gen2 

3. There are 2 files. delivery and tracking. how the folder structure be best. [Seperate or same folder]. i have created seperate, but not sure if it is a best practise.

Please help

Thanks a lot