topic Re: Automate the Databricks workflow deployment in Data Engineering

Automate the Databricks workflow deployment

joao_vnb — Wed, 08 Feb 2023 15:36:02 GMT

Hi everyone,

Do you guys know if it's possible to automate the Databricks workflow deployment through azure devops (like what we do with the deployment of notebooks)?

Re: Automate the Databricks workflow deployment

Manoj12421 — Thu, 09 Feb 2023 19:27:32 GMT

This might help you:-

https://learn.microsoft.com/en-us/azure/databricks/dev-tools/ci-cd/ci-cd-azure-devops

Re: Automate the Databricks workflow deployment

Anonymous — Sat, 11 Feb 2023 05:26:07 GMT

To automate the deployment of Databricks workflows, you can use the Databricks REST API and a scripting language such as Python or Bash. The script can create a new workflow and add steps to it, as well as manage existing workflows. The script can be triggered by a CI/CD pipeline, allowing for automated deployment of the workflow whenever changes are pushed to the source code repository. This approach can greatly simplify and streamline the deployment process, reducing the risk of manual errors and improving the speed of delivery.

Re: Automate the Databricks workflow deployment

joao_vnb — Tue, 14 Feb 2023 14:01:10 GMT

thanks for the answer! could you give an example of the structure of the python script?

or a documentation link where I can find it...

Re: Automate the Databricks workflow deployment

joao_vnb — Tue, 14 Feb 2023 14:05:00 GMT

thanks for the answer!! It isn't exactly what I want, though

Re: Automate the Databricks workflow deployment

joao_vnb — Tue, 14 Feb 2023 14:05:18 GMT

thanks for the answer!! It isn't exactly what I want, though

Re: Automate the Databricks workflow deployment

Debayan — Fri, 10 Feb 2023 05:29:13 GMT

Hi, yes, it is possible. Please refer: https://learn.microsoft.com/en-us/azure/databricks/dev-tools/ci-cd/ci-cd-azure-devops and let us know if this helps.

Re: Automate the Databricks workflow deployment

asingamaneni — Mon, 12 Feb 2024 18:46:13 GMT

Did you get a chance to try Brickflows - https://github.com/Nike-Inc/brickflow

You can find the documentation here - https://engineering.nike.com/brickflow/v0.11.2/

Brickflow uses - Databricks Asset Bundles(DAB) under the hood but provides a Pythonic way to create your workflows.

Re: Automate the Databricks workflow deployment

Manoranjan — Wed, 02 Jul 2025 14:07:47 GMT

Hi,

Above link does not work. Could you please provide the appropriate link