Hey @athang 

If your data ecosystem is already centered around Spark, Delta Lake, etc., then Databricks is a great fit — especially with Unity Catalog, which is really the heart of the platform.

Governance becomes an integral part of the system — you don’t have to put in extra effort to manage it separately.

For example, in our setup:

• We ingest high-frequency data into Kafka

• From there:

  • Ingestion into the RAW layer

  • Transformation into structured tables

  • Workflows / Jobs orchestration

• Everything is organized under catalog.schema (Tables | Models | Volumes)

• User access management

• Job compute clusters and all-purpose interactive clusters for development

• Dashboards for observability and data quality metrics

• ML lifecycle management and model serving

You name it — it’s covered, and everything is governed through Unity Catalog.

We manage infrastructure as code using Terraform
and deployments through Databricks Asset Bundles.

This solution — including governance — also works perfectly with externally managed locations.