topic DevOps CICD for Databricks Schema in Community Articles

DevOps CICD for Databricks Schema

Mot — Thu, 18 Jul 2024 02:58:13 GMT

Does anyone know how to do DevOps CICD pipeline to deploy changes to a Databricks Catalog? I've added a couple tables in DEV and need to deploy in Prod. Could'nt find any info on this anywhere.

Re: DevOps CICD for Databricks Schema

Mot — Fri, 19 Jul 2024 22:43:29 GMT

Thanks @Retired_mod , the links refer mostly deployments of workspace objects rather than changes in database objects, eg. a new schema, table, or new column added to a table etc. I'll keep looking...

Re: DevOps CICD for Databricks Schema

szymon_dybczak — Sat, 20 Jul 2024 02:37:21 GMT

Hi @Mot ,

You're looking for state based schema migration. You can achieve this using external tools like liquibase

https://medium.com/dbsql-sme-engineering/advanced-schema-management-on-databricks-with-liquibase-1900e9f7b9c0

Also take a look on below discussion at reddit, they are mentioning terraform as well:

https://www.reddit.com/r/dataengineering/s/k5ng2Qw84T