Usage of Databricks apps or UI driven approach to create & maintain Databricks infrastructure

[noorbasha534]

Hi all,

the CI/CD based process to create & maintain Databricks infrastructure (UC securables, Metastore securables, Workspace securables) is resulting into high time to market in our case. So, we are planning to make it UI driven, as on, create a Databricks app to create and maintain the securables.

We plan to persist all the config in delta tables.

Have you seen someone doing such thing or what do you think about such an implementation pattern..

appreciate the mindshare

[szymon_dybczak]

Hi @noorbasha534 ,

To be honest, I've never seen this approach before. In most cases, in production environments, you’ll encounter CI/CD pipelines in Terraform that provision all the infrastructure for Databricks, along with Databricks Asset Bundles CI/CD pipelines responsible for deploying code, workflows, DLT, etc. But if your organization finds it more beneficial to move away from that approach, you should give it a try. Maybe in your case, this method will be completely sufficient. 🙂

[Pat]

Hi, this doesn't sound good to me TBH.
I don't know why this is high in time, maybe try to look at some modules/examples to create resources via terraform, ie.:
https://github.com/databricks/terraform-databricks-examples
https://github.com/databricks/terraform-databricks-sra

I have done this few times from scratch, you setup re-usable modules then you are ready to use them basically.

When to use DABs vs Terraform: https://medium.com/@alexott_en/terraform-vs-databricks-asset-bundles-6256aa70e387

Maybe I didn't get your problem, but this shouldn't be time consuming TBH.

[nayan_wylde]

UI Driven approach is definetly a bad idea for deployment. I have seen most organisation using terraform or biceps for deployment.

Why UI-Driven Infrastructure is Wrong

1. No version control or audit trail

2. Configuration Drift & Inconsistency

3. No Disaster Recovery Strategy

4. Security & Compliance Issues.

The table will explain:

Scenario                                       UI Approach                                   Terraform Approach

Deploy to 3 environments	2-3 hours manual work	5 minutes automated
Add security policy	Update each workspace manually	Change 1 line, apply everywhere
Disaster recovery	Days to weeks	Minutes
Audit compliance	Screenshots and docs	Git history + automated reports
Team onboarding	Training sessions + documentation	Read the code
Change approval	Email chains + manual verification	Pull requests + automated validation

The Right Way: Infrastructure as Code

1. Terraform (Recommended)

2. Biceps (Azure-Specific)

3. Asset Bundle (Complex Logic)
4. Pulumi (Complex Logic)