How do I kick off Azure Data Factory from within Databricks?

[StephanieAlba]

I want to kick off ingestion in ADF from Databricks. When ADF ingestion is done, my DBX bronze-silver-gold pipeline follows within DBX.

I see it is possible to call Databricks notebooks from ADF. Can I also go the other way? I want to start the ingestion and then handle the rest of my pipeline in Workflows. Does anyone have an example of doing this?

[Hubert-Dudek · Accepted solution]

You can create an Azure logic app that will call Azure Data Factory (there, you can choose from ready actions).

Azure logic app as a trigger can have web requests (REST API).

You can invoke it at the end of the notebook by calling a request. (requests.post(url))

You can also easily handle authentication by adding key headers and comparing in the logic app.

[PriyaAnanthram]

hmmm the usecase is You want databricks to run its pipeline be notified then run the rest of your code in ADF

So as hubert mentioned you can have a logic app that calls ADF and databricks calls the logic app using the http trigger. Another way I can think of is calling the ADF directly using the REST endpoint Pipelines - Create Run - REST API (Azure Data Factory) | Microsoft Learn

[StephanieAlba]

No, I don't think I was clear enough. I want to kick off ingestion in ADF from Databricks. When ADF ingestion is done, my DBX pipeline follows within DBX.

[-werners-]

I never did this (only the other way around), but what I do know is that you can launch ADF pipelines using the ADF API. So you could send a rest call.

However this is async, so you will have to poll (at least the last time I checked).

[KKo]

Are you looking to pass output of databricks notebook to ADF?

[StephanieAlba]

no