Hey @athang
If your data ecosystem is already centered around Spark, Delta Lake, etc., then Databricks is a great fit — especially with Unity Catalog, which is really the heart of the platform.
Governance becomes an integral part of the system — you don’t have to put in extra effort to manage it separately.
For example, in our setup:
We ingest high-frequency data into Kafka
From there:
Ingestion into the RAW layer
Transformation into structured tables
Workflows / Jobs orchestration
Everything is organized under catalog.schema (Tables | Models | Volumes)
User access management
Job compute clusters and all-purpose interactive clusters for development
Dashboards for observability and data quality metrics
ML lifecycle management and model serving
You name it — it’s covered, and everything is governed through Unity Catalog.
We manage infrastructure as code using Terraform
and deployments through Databricks Asset Bundles.
This solution — including governance — also works perfectly with externally managed locations.