Azure Devops CI/CD - AWS Databricks

[thiagoawstest]

Hello, there is documentation for integrating Azure Devops CI/CD pipeline with AWS Databricks.

Thanks.

[jacovangelder]

What are you trying to achieve exactly? Deploy asset bundles or provision Databricks infrastructure of some sort? 

[thiagoawstest]

Hi, today I already have a CI/CD pipeline between Azure Devops + Azure Databricks, now I need to integrate my Azure Devops with AWS Databricks, to run a CI/CD pipeline, this is what I want to achieve.

[jacovangelder]

But what does the CI/CD pipeline do, deploy notebooks/asset bundles/provision Databricks workspace(s)?
You should be able to authenticate to databricks using the Databricks cli or API regardless of the CI tool you're using. 

[thiagoawstest]

Yes, it implements notebooks, jobs, etc....

But the integration of Azure Devops would be done via token, service principal, what is the best practice, is there any documentation?, because I don't want to define a personal token, if the user is removed, the integration will stop.

[jacovangelder · Accepted solution]

You'll need to install the AWS Toolkit in Azure DevOps, that way you can make a service connection inside your Azure DevOps project that authenticates using an AWS Access Key ID/Secret Access Key (the AWS equivalent to Azure Service Principals). Hope this helps! https://aws.amazon.com/vsts/

[thiagoawstest]

I understood that the integration would be direct with Databricks, but I will check.

Thank you for your help.