topic Re: Azure Devops CI/CD - AWS Databricks in Data Engineering

Azure Devops CI/CD - AWS Databricks

thiagoawstest — Thu, 13 Jun 2024 19:53:46 GMT

Hello, there is documentation for integrating Azure Devops CI/CD pipeline with AWS Databricks.

Thanks.

Re: Azure Devops CI/CD - AWS Databricks

jacovangelder — Fri, 14 Jun 2024 07:12:16 GMT

What are you trying to achieve exactly? Deploy asset bundles or provision Databricks infrastructure of some sort?

Re: Azure Devops CI/CD - AWS Databricks

thiagoawstest — Fri, 14 Jun 2024 12:34:27 GMT

Hi, today I already have a CI/CD pipeline between Azure Devops + Azure Databricks, now I need to integrate my Azure Devops with AWS Databricks, to run a CI/CD pipeline, this is what I want to achieve.

Re: Azure Devops CI/CD - AWS Databricks

jacovangelder — Fri, 14 Jun 2024 12:39:18 GMT

But what does the CI/CD pipeline do, deploy notebooks/asset bundles/provision Databricks workspace(s)?
You should be able to authenticate to databricks using the Databricks cli or API regardless of the CI tool you're using.

Re: Azure Devops CI/CD - AWS Databricks

thiagoawstest — Fri, 14 Jun 2024 13:11:48 GMT

Yes, it implements notebooks, jobs, etc....

But the integration of Azure Devops would be done via token, service principal, what is the best practice, is there any documentation?, because I don't want to define a personal token, if the user is removed, the integration will stop.

Re: Azure Devops CI/CD - AWS Databricks

jacovangelder — Fri, 14 Jun 2024 13:27:34 GMT

You'll need to install the AWS Toolkit in Azure DevOps, that way you can make a service connection inside your Azure DevOps project that authenticates using an AWS Access Key ID/Secret Access Key (the AWS equivalent to Azure Service Principals). Hope this helps! https://aws.amazon.com/vsts/

Re: Azure Devops CI/CD - AWS Databricks

thiagoawstest — Fri, 14 Jun 2024 13:50:48 GMT

I understood that the integration would be direct with Databricks, but I will check.

Thank you for your help.