cancel
Showing results for 
Search instead for 
Did you mean: 
Knowledge Sharing Hub
Dive into a collaborative space where members like YOU can exchange knowledge, tips, and best practices. Join the conversation today and unlock a wealth of collective wisdom to enhance your experience and drive success.
cancel
Showing results for 
Search instead for 
Did you mean: 

DevOps CICD for Databricks Schema

Mot
New Contributor II

Does anyone know how to do DevOps CICD pipeline to deploy changes to a Databricks Catalog? I've added a couple tables in DEV and need to deploy in Prod. Could'nt find any info on this anywhere. 

1 ACCEPTED SOLUTION

Accepted Solutions

szymon_dybczak
Esteemed Contributor III

Hi @Mot ,

You're looking for state based schema migration. You can achieve this using external tools like liquibase

https://medium.com/dbsql-sme-engineering/advanced-schema-management-on-databricks-with-liquibase-190...

Also take a look on below discussion at reddit, they are mentioning terraform as well:

https://www.reddit.com/r/dataengineering/s/k5ng2Qw84T

View solution in original post

2 REPLIES 2

Mot
New Contributor II

Thanks @Retired_mod , the links refer mostly deployments of workspace objects rather than changes in database objects, eg. a new schema, table, or new column added to a table etc. I'll keep looking...

 

szymon_dybczak
Esteemed Contributor III

Hi @Mot ,

You're looking for state based schema migration. You can achieve this using external tools like liquibase

https://medium.com/dbsql-sme-engineering/advanced-schema-management-on-databricks-with-liquibase-190...

Also take a look on below discussion at reddit, they are mentioning terraform as well:

https://www.reddit.com/r/dataengineering/s/k5ng2Qw84T

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group