Hi all.

If you've ever manually promoted resources from dev to prod on Databricks — copying notebooks, updating configs, hoping nothing breaks — this post is for you.

I've been building a CI/CD setup for a Speech-to-Text pipeline on Databricks, and I wanted to share the approach in case it's useful to others here. The goal was simple: treat Databricks resources as code, deploy them deterministically across environments, and authenticate from GitHub Actions without storing any long-lived tokens.

The stack is:

• Databricks Asset Bundles for infrastructure-as-code
• GitHub Actions for delivery
• OIDC federation for authentication.

What gets deployed by the bundle

The bundle manages the full solution end-to-end:

• Unity Catalog schema and volume — created automatically on deploy, no manual setup
• Silver pipelines (Spark Declarative Pipelines) — audio ingestion via Auto Loader and NLP enrichment with two parallel implementations: AI SQL functions and Foundation Model API
• Gold tables — transcription output from Whisper Large V3 via Model Serving endpoint, plus NLP evaluation results tracked with MLflow
• Model Serving endpoint — Whisper Large V3 for audio transcription
• AI/BI Dashboard — monitoring transcription quality and NLP results
• Genie Space — deployed as a job, since direct bundle support isn't available yet; it's a workaround worth knowing about if you're hitting the same limitation
• Orchestration job (stt_main) — sequences all the stages in order

Everything lives in the bundle YAML. If it's not in the repo, it doesn't exist in the workspace.

What the CI/CD setup covers

• Structuring the repo with the bundle isolated from application code
• Declaring dev and prod environments using DAB targets — same bundle YAML, different configurations
• Configuring a service principal with minimal Unity Catalog permissions
• Setting up OIDC federation policies so GitHub Actions authenticates without PATs
• GitHub Environments to isolate variables and secrets per environment, with required reviewers on Prod
• A workflow that runs bundle validate → bundle plan → bundle deploy, passing service_principal_id as the only external variable

One thing worth calling out: in dev, the workflow also syncs a Git folder in the workspace before deploying — useful for interactive development. In prod, the bundle is the only source of truth and the Git folder sync doesn't happen.

Resources

The full repo is on GitHub — the GitHub Actions workflows and all the DAB configuration with inline comments on every step are there: 🔗 https://github.com/alessandro9110/Speech-To-Text-With-Databricks

If you want the full walkthrough with context and explanation behind each decision, I wrote a detailed article on Medium: 🔗 https://medium.com/towards-data-engineering/ci-cd-on-databricks-with-asset-bundles-and-github-action...

Happy to answer questions or discuss alternative approaches — particularly around multi-workspace setups, how to handle Unity Catalog permissions when the deploy identity differs from run_as, or the Genie workaround if you're dealing with the same limitation.

Thank you to everyone for the support ❤️