topic Re: What are some best practices for CICD? in Data Engineering

What are some best practices for CICD?

Anonymous — Mon, 07 Jun 2021 17:50:07 GMT

A number of people have questions on using Databricks in a productionalized environment. What are the best practices to enable CICD automation?

Re: What are some best practices for CICD?

Anonymous — Mon, 07 Jun 2021 17:51:47 GMT

Databricks enables CICD using REST API. That allows build servers (such as Jenkins, github actions, etc) to update artifacts.

In addition one should use CICD practices to store databricks job config. Use the REST API to update and refresh those in the appropriate environment.

https://docs.databricks.com/dev-tools/api/latest/index.html

Re: What are some best practices for CICD?

gbrueckl — Fri, 03 Sep 2021 06:39:34 GMT

If you use PowerShell for your CI/CD pipelines you may want to have a look at my DatabricksPS module which is a wrapper for the Databricks REST API.

It has dedicated cmdlets to export (e.g. from DEV) and import again (e.g. to TEST/PROD) in an automated fashion. This includes notebooks, job-definitions, cluster-definitions, secrets, ..., (and SQL objects to come soon!)

https://www.powershellgallery.com/packages/DatabricksPS

Re: What are some best practices for CICD?

pawelmitrus — Tue, 14 Sep 2021 09:11:22 GMT

Some time ago I've been looking for the very same answers and this is what I found/did back then:

https://menziess.github.io/howto/ - my source of inspiration (tech part)
https://databricks.com/blog/2017/10/30/continuous-integration-continuous-delivery-databricks.html - based on that I've been trying to set up some process-wise baseline
https://github.com/pawelmitrus/azure-databricks-cicd-notebooks - my own idea of solving the problem of CI/CD for the very common projects, using notebooks for running ETLs (it's kind of a mix of all above)

I'd be happy to discuss

Re: What are some best practices for CICD?

Kristian_Schnei — Tue, 14 Sep 2021 10:06:42 GMT

I don't know if it's best practice, but perhaps it can serve as inspiration.

We do CI/CD with unit test of pyspark code with github actions. Have a look:

https://github.com/Energinet-DataHub/geh-aggregations#databricks-workspace

https://github.com/Energinet-DataHub/geh-aggregations/actions/workflows/aggregation-job-infra-cd.yml

Re: What are some best practices for CICD?

MadelynM — Mon, 08 Nov 2021 18:38:21 GMT

Check out the Databricks Labs CI/CD Templates. This repository provides a template for automated Databricks CI/CD pipeline creation and deployment.

Table of Contents

Re: What are some best practices for CICD?

User16859945835 — Tue, 09 Nov 2021 05:21:03 GMT

Two additional resources come to mind:

If using Jenkins, there's a best practice guide for CI/CD using Jenkins that was written based on numerous successful implementations.
There is an on-demand webinar focused on Getting Workloads to Production from a DevOps and CI/CD perspective.

Re: What are some best practices for CICD?

alexott — Thu, 25 Nov 2021 18:44:05 GMT

there is no such REST API (yet)

Re: What are some best practices for CICD?

Atanu — Sat, 27 Nov 2021 14:57:24 GMT

We have DBFS APIs, but not sure if that solve your purpse @Erik Parmann . https://docs.databricks.com/dev-tools/api/latest/dbfs.html#list you can check out this.

Re: What are some best practices for CICD?

gbrueckl — Mon, 29 Nov 2021 08:18:22 GMT

I guess one could say the same for all the SQL meta objects for which you also need to have a cluster up and running

but some just need a cluster up and running to validate and check them

some even rely on a cluster e.g. for authentication - if you used a mount point with OAuth for example

Re: What are some best practices for CICD?

LorenRD — Wed, 17 Aug 2022 11:22:14 GMT

Hello there, I would like to retake this thread to ask for good practices for Data processes in Databricks. I have 2 cloud accounts with one Databricks env in each one (One for dev another for prod).

I was thinking to create my own CI/CD pipeline to move notebooks from dev env to prod and schedule them with GitHub and Azure DevOps but would like to see what community recommends.

I've seen that ci-cd templates is deprecated and now is recommended to use dbx, is it a tool for that purpose?

Re: What are some best practices for CICD?

xiangzhu — Fri, 18 Nov 2022 15:12:27 GMT

https://github.com/databrickslabs/cicd-templates is legacy now, the updated one here: dbx.

you need to walkthrough the doc carefully, there're many informations inside, and BTW `dbx init` could create a template.

Re: What are some best practices for CICD?

Erik — Thu, 11 Nov 2021 14:41:58 GMT

We are using the databricks terraform provider to handle... everything really. Then we use a CI runner (in our case azure pipelines) to deploy to dev/test/prod depending on branches and stuff in git (you might prefer tags/branches whatever your branching strategy is).

It works pretty good, EXCEPT, validating mounts take a long time (10-15 min) because it needs to spin up a cluster. That is pretty lame, and the only fix seems to be for databricks to make a REST API letting you list/modify mounts, but this is nowhere on any list.

Re: What are some best practices for CICD?

Erik — Sun, 28 Nov 2021 21:33:18 GMT

(Note: The mount issue is not mine alone, its a problem for everyone using the terraform databricks provider )

I guess one could use DBFS api to determine if anything was present at the mount-point, but it wont tell you if its because something is actually mounted there, or where it is mounted from. So one would still have to start a cluster to check those things:-/

Re: What are some best practices for CICD?

Erik — Sun, 28 Nov 2021 21:34:11 GMT

Unfortunately not. Where do I vote to make it happen faster?

Re: What are some best practices for CICD?

BaivabMohanty — Wed, 24 Apr 2024 15:49:08 GMT

Any leads/posts for Databricks CI/CD integration with Bitbucket pipeline. I am facing the below error while I creation my CICD pipeline

pipelines:
branches:
master:
- step:
name: Deploy Databricks Changes
image: docker:19.03.12
services:
- docker
script:
# Update and install required packages
- apk add --no-cache py-pip expect git
- pip install --upgrade pip
- pip install databricks-cli
- databricks --version

# Create Databricks CLI configuration file for Source Workspace
- echo -e "[DEFAULT]\nhost = $DB_HOST_SOURCE\ntoken = $DB_TOKEN_QA" > ~/.databrickscfg_source
# Create Databricks CLI configuration file for Destination Workspace
- echo -e "[DEFAULT]\nhost = $DB_HOST_DESTINATION\ntoken = $DB_TOKEN_PROD" > ~/.databrickscfg_destination

# Display the contents of the configuration files for verification
- cat ~/.databrickscfg_source
- cat ~/.databrickscfg_destination

# Configure Databricks CLI for source
- |
expect -c "
spawn /usr/local/bin/databricks configure --host $DB_HOST_SOURCE --token $DB_TOKEN_QA
expect eof
"

# Configure Databricks CLI for destination
- |
expect -c "
spawn /usr/local/bin/databricks configure --host $DB_HOST_DESTINATION --token $DB_TOKEN_PROD
expect eof
"

# Clone the latest code from Master branch using App Password
- git clone --depth 1 https://<>username:<token>@bitbucket.org/ttec-digital-ip/insights_databricks_qa.git /tmp/source_workspace_export/

# Configure Databricks CLI with token for destination using expect
- |
expect -c "
spawn /usr/local/bin/databricks configure --profile destination
expect \"Databricks Host (should begin with https://): \"
send \"$DB_HOST_DESTINATION\r\"
expect \"Token: \"
send \"$DB_TOKEN_PROD\r\"
expect eof
"

# Copy files from Local File System to Destination Workspace
- databricks fs cp -r /tmp/source_workspace_export /Workspace/Shared/ --profile destination

Error: InvalidConfigurationError: You haven't configured the CLI yet for the profile destination! Please configure by entering `/usr/bin/databricks configure --profile destination`