cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

How do I kick off Azure Data Factory from within Databricks?

StephanieAlba
Valued Contributor III

I want to kick off ingestion in ADF from Databricks. When ADF ingestion is done, my DBX bronze-silver-gold pipeline follows within DBX.

I see it is possible to call Databricks notebooks from ADF. Can I also go the other way? I want to start the ingestion and then handle the rest of my pipeline in Workflows. Does anyone have an example of doing this?

1 ACCEPTED SOLUTION

Accepted Solutions

Hubert-Dudek
Esteemed Contributor III

You can create an Azure logic app that will call Azure Data Factory (there, you can choose from ready actions).

Azure logic app as a trigger can have web requests (REST API).

You can invoke it at the end of the notebook by calling a request. (requests.post(url))

You can also easily handle authentication by adding key headers and comparing in the logic app.

View solution in original post

6 REPLIES 6

Hubert-Dudek
Esteemed Contributor III

You can create an Azure logic app that will call Azure Data Factory (there, you can choose from ready actions).

Azure logic app as a trigger can have web requests (REST API).

You can invoke it at the end of the notebook by calling a request. (requests.post(url))

You can also easily handle authentication by adding key headers and comparing in the logic app.

PriyaAnanthram
Contributor III

hmmm the usecase is You want databricks to run its pipeline be notified then run the rest of your code in ADF

So as hubert mentioned you can have a logic app that calls ADF and databricks calls the logic app using the http trigger. Another way I can think of is calling the ADF directly using the REST endpoint Pipelines - Create Run - REST API (Azure Data Factory) | Microsoft Learn

No, I don't think I was clear enough. I want to kick off ingestion in ADF from Databricks. When ADF ingestion is done, my DBX pipeline follows within DBX.

-werners-
Esteemed Contributor III

I never did this (only the other way around), but what I do know is that you can launch ADF pipelines using the ADF API. So you could send a rest call.

However this is async, so you will have to poll (at least the last time I checked).

KKo
Contributor III

Are you looking to pass output of databricks notebook to ADF?

StephanieAlba
Valued Contributor III

no

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group