cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

How do I kick off Azure Data Factory from within Databricks?

StephanieRivera
Valued Contributor II
Valued Contributor II

I want to kick off ingestion in ADF from Databricks. When ADF ingestion is done, my DBX bronze-silver-gold pipeline follows within DBX.

I see it is possible to call Databricks notebooks from ADF. Can I also go the other way? I want to start the ingestion and then handle the rest of my pipeline in Workflows. Does anyone have an example of doing this?

1 ACCEPTED SOLUTION

Accepted Solutions

Hubert-Dudek
Esteemed Contributor III

You can create an Azure logic app that will call Azure Data Factory (there, you can choose from ready actions).

Azure logic app as a trigger can have web requests (REST API).

You can invoke it at the end of the notebook by calling a request. (requests.post(url))

You can also easily handle authentication by adding key headers and comparing in the logic app.

View solution in original post

7 REPLIES 7

Hubert-Dudek
Esteemed Contributor III

You can create an Azure logic app that will call Azure Data Factory (there, you can choose from ready actions).

Azure logic app as a trigger can have web requests (REST API).

You can invoke it at the end of the notebook by calling a request. (requests.post(url))

You can also easily handle authentication by adding key headers and comparing in the logic app.

PriyaAnanthram
Contributor III

hmmm the usecase is You want databricks to run its pipeline be notified then run the rest of your code in ADF

So as hubert mentioned you can have a logic app that calls ADF and databricks calls the logic app using the http trigger. Another way I can think of is calling the ADF directly using the REST endpoint Pipelines - Create Run - REST API (Azure Data Factory) | Microsoft Learn

No, I don't think I was clear enough. I want to kick off ingestion in ADF from Databricks. When ADF ingestion is done, my DBX pipeline follows within DBX.

-werners-
Esteemed Contributor III

I never did this (only the other way around), but what I do know is that you can launch ADF pipelines using the ADF API. So you could send a rest call.

However this is async, so you will have to poll (at least the last time I checked).

KKo
Contributor III

Are you looking to pass output of databricks notebook to ADF?

StephanieRivera
Valued Contributor II
Valued Contributor II

no

Kaniz
Community Manager
Community Manager

Hi @Stephanie Rivera​​, We haven’t heard from you since the last response from @Werner Stinckens​ , and I was checking back to see if you have a resolution yet.

If you have any solution, please share it with the community as it can be helpful to others. Otherwise, we will respond with more details and try to help.

Also, Please don't forget to click on the "Select As Best" button whenever the information provided helps resolve your question.

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.