topic Re: How do I kick off Azure Data Factory from within Databricks? in Data Engineering

How do I kick off Azure Data Factory from within Databricks?

StephanieAlba — Wed, 12 Oct 2022 16:49:41 GMT

I want to kick off ingestion in ADF from Databricks. When ADF ingestion is done, my DBX bronze-silver-gold pipeline follows within DBX.

I see it is possible to call Databricks notebooks from ADF. Can I also go the other way? I want to start the ingestion and then handle the rest of my pipeline in Workflows. Does anyone have an example of doing this?

Re: How do I kick off Azure Data Factory from within Databricks?

Hubert-Dudek — Wed, 12 Oct 2022 16:55:08 GMT

You can create an Azure logic app that will call Azure Data Factory (there, you can choose from ready actions).

Azure logic app as a trigger can have web requests (REST API).

You can invoke it at the end of the notebook by calling a request. (requests.post(url))

You can also easily handle authentication by adding key headers and comparing in the logic app.

Re: How do I kick off Azure Data Factory from within Databricks?

PriyaAnanthram — Wed, 12 Oct 2022 22:39:06 GMT

hmmm the usecase is You want databricks to run its pipeline be notified then run the rest of your code in ADF

So as hubert mentioned you can have a logic app that calls ADF and databricks calls the logic app using the http trigger. Another way I can think of is calling the ADF directly using the REST endpoint Pipelines - Create Run - REST API (Azure Data Factory) | Microsoft Learn

Re: How do I kick off Azure Data Factory from within Databricks?

StephanieAlba — Wed, 12 Oct 2022 23:26:45 GMT

No, I don't think I was clear enough. I want to kick off ingestion in ADF from Databricks. When ADF ingestion is done, my DBX pipeline follows within DBX.

Re: How do I kick off Azure Data Factory from within Databricks?

KKo — Thu, 13 Oct 2022 03:59:36 GMT

Are you looking to pass output of databricks notebook to ADF?

Re: How do I kick off Azure Data Factory from within Databricks?

-werners- — Thu, 13 Oct 2022 07:08:19 GMT

I never did this (only the other way around), but what I do know is that you can launch ADF pipelines using the ADF API. So you could send a rest call.

However this is async, so you will have to poll (at least the last time I checked).

Re: How do I kick off Azure Data Factory from within Databricks?

StephanieAlba — Thu, 13 Oct 2022 17:06:13 GMT