FinOps at Scale: Building a Repeatable Architecture for Cross-Account Cost Visibility in Databricks

[mreiling_data]

In today’s enterprise data landscape, large organizations often operate multiple Databricks workspaces across cloud accounts, regions, and business units. While this flexibility enables autonomy and scalability, it also creates a major challenge for FinOps, cost reporting, and central governance teams: how do you gain unified visibility into spend across all environments without introducing significant operational overhead or custom development?

Cross-account cost reporting is no longer a nice-to-have. It is table stakes for mature cloud financial operations. Yet many teams still rely on fragmented, manual, or ad hoc processes to track and aggregate costs across their Databricks estate. That slows decision-making, limits cost accountability, and introduces unnecessary risk.

Databricks provides built-in usage dashboards that give customers a strong starting point for monitoring account- and workspace-level usage. These dashboards are useful for platform and account admins who need visibility into usage by workspace, SKU, and tags within a given account. But they have limitations for customers operating across multiple accounts and multiple clouds, where teams often need to stitch together reporting across account-level views to get a more unified picture. For platform admins, the current setup can still work well for local account-level analysis. For FinOps teams, however, the need is different: they want single-pane-of-glass usage insights across the broader estate.

In this blog, we’ll walk through two distinct approaches to building a repeatable, best-practices-driven architecture for cross-account cost aggregation in Databricks. This setup supplements the existing usage dashboards by allowing you to roll up local account-level reporting into a broader cross-account view, rather than replacing the native experience entirely. Whether you’re managing five workspaces across multiple clouds or fifty, this approach can help you centralize cost and usage insights, reduce manual reporting overhead, and make that data more actionable for teams beyond platform administration.

• Standardize cost reporting across clouds and accounts
• Enable cost transparency and chargeback models
• Empower FinOps teams with near real-time trusted data
• Unlock deeper insights into spend by workspace, user, or workload

Let’s dive into the architecture, tooling, and best practices that make this possible. Both approaches rely on Unity Catalog for the centralized metadata and user management needed to support cross-workspace security and sharing.

Prerequisites

• UC Metastore + UC-enabled workspaces
• Delta Sharing enabled (Push Model only)
• Supported compute for governance: DBR 16.4+ or Serverless
• Access to system tables in each workspace

Sharing Approach (Push Model)

This model is suited to customers who want to offload aggregation compute to the source workspaces and keep the central FinOps environment lightweight. Each source account runs its own jobs to aggregate billing data locally, then uses Delta Sharing to securely publish only curated usage tables to the central FinOps workspace. This avoids moving large raw datasets, reduces central compute and networking overhead, and fits organizations that prioritize per-account autonomy over transformation logic and scheduling while still needing a unified central view.

Architecture overview:

Key implementation steps:

1. Create a central FinOps reporting workspace attached to your existing Unity Catalog metastore, with access to the relevant system tables (via account admin or FGAC-scoped policies, depending on who runs the jobs).
2. Automate monthly usage export per account from one workspace in each account (any cloud) using automation scripts and service principals with access to the billing system tables and compute in that workspace. These scripts should aggregate the data and create a Delta share, where the share represents the aggregated output rather than the system tables themselves.
3. Run a monthly Lakeflow job in the FinOps workspace that reads and aggregates all Delta shares into curated Delta tables in a central FinOps schema.
4. Expose the data per-team using FGAC (for example, ABAC/row-level filters). Add a simple team mapping to associate usage with teams or tenants and apply FGAC so users can query the same table while only seeing rows that are relevant to them.

Suggested naming pattern: Use a central schema such as finops_core with curated tables like finops_core.fact_usage_monthly and a team mapping table like finops_core.dim_team. Use finops_core.fact_usage_monthly as the shared fact table, securing access via ABAC/row-level filters directly on the shared table (FGAC). Optionally, add per-team schemas such as finops_marketing with views like finops_marketing.cost_by_month for convenience, while relying on FGAC on the shared table for authorization.

Federation Approach (Pull Model)

This model is ideal for customers who want central control over cost data extraction and governance and need flexible, cross-workspace analysis. Using Lakehouse Federation, a central workspace can query billing data across all workspaces through a unified UNION view for ad-hoc exploration, then orchestrate materialization into a governed hub table. This suits organizations that prefer one centrally managed pipeline and policy surface (ABAC/RLS), with a single place to troubleshoot, enrich, and secure FinOps data for diverse consumers.

Architecture overview:

Key implementation steps:

1. Leverage Lakehouse Federation to create connections to compute resources in each source workspace.
2. Create a central schema containing tables for mapping of accounts and additional data such as adjustments to list prices.
   • If you’d like to query ad-hoc, create a view that’s a union of all billing tables.
3. Schedule a job to extract data from each federated catalog and store the data in a central hub table.
4. Apply fine-grained access control on the central table (for example, leveraging ABAC/row-level filters), so users can only query the slices of data they’re permitted to see.
5. Generate a daily extract to be fed to downstream systems that require this information.

Suggested naming pattern: Use federated catalogs such as fed_ws_marketing that expose each workspace’s system.billing.usage table, a central schema such as finops_core with tables like finops_core.account_map and finops_core.price_adjustments, an ad hoc union view like finops_core.v_billing_all_workspaces, and a central hub table like finops_core.fact_usage_daily. For downstream exports, use an exports schema and table such as finops_exports.daily_finops_snapshot.

Get Started

Accelerate your FinOps rollout with a quickstart for unified cost reporting across regions, accounts, and clouds. Deploy the hub, aggregate data from spokes, and build your first reporting dashboard - all using the Databricks open source AI Dev Kit to bootstrap your environment, generate Databricks resources, and move from concept to FinOps at scale faster.