How to bring the project id info into the databricks billing usage table on GCP

[Danish11052000]

Hi,

We are using Databricks on GCP and are currently analyzing costs using the billing usage/system tables.

One challenge we are facing is that the billing usage data provides usage and cost information, but we are unable to identify the associated GCP Project ID for each usage record.

Our goal is to:

• Map Databricks usage and costs to specific GCP projects.
• Generate project-level chargeback/showback reporting.
• Understand which GCP project is consuming the most Databricks resources.

Has anyone implemented a solution for this?

Any guidance, examples, or best practices would be greatly appreciated.

[Satyasai]

The reason you are not seeing the GCP Project ID directly in the system.billing.usage table is that Databricks system tables track DBU usage at the workspace, cluster, and workload level rather than underlying Google Cloud project infrastructure.

As such, Databricks workspaces in GCP are typically deployed either within a specific customer-managed VPC / GCP Project or across multiple projects and teams successfully implement showback/chargeback using a few proven strategies.

Solution 1: Workspace to Project Mapping Table (Easiest)

If your architecture utilizes specific Databricks Workspaces to specific GCP Projects (i.e., Project A hosts Workspace A, Project B hosts Workspace B), create a reference mapping table in Unity Catalog.

Create a Mapping Table and Join with System Billing Usage

[balajij8]

@Danish11052000 System tables does not generally carry details such as GCP Project ID. You can create an static mapping table (GCP Project id, other details, workspace id) in Unity Catalog and use the info with the system tables if your deployment is like every Databricks workspace maps to exactly one GCP project. You can load this mapping using Terraform deployment scripts, since each workspace's resources land in a specific project. You can join the lookup table to system.billing.usage on workspace_id, filtering for cloud = 'GCP' and record_type = 'ORIGINAL'. You can also join system.billing.list_prices to convert the DBUs to dollars, giving the correct project level chargeback report to see which GCP project is consuming the most resources.

You can rely on sub-workspace attribution via custom tags if you have multiple GCP projects sharing a single workspace. You can apply a custom tag like gcp_project_id = <project> to the clusters, SQL warehouses and pools so Databricks propagates it into the custom_tags column of system.billing.usage. To prevent untagged usage, enforce this through cluster policies so computes cannot even be created without it. For the serverless, you can leverage serverless usage policies to auto-tag the usage and it lands in the custom_tags column.

You can extract custom_tags['gcp_project_id'] in the queries for the per-project cost breakdowns and that tags only apply to usage incurred after the tag is set - old records will remain untagged and hence get these policies in place quickly.

[data_pulse]

@Danish11052000 

A feasible workaround is to use the Databricks Account Workspaces API to fetch the GCP project_id alongside workspace_id. The workspace API schema exposes it under cloud_resource_container.gcp.project_id. Reference

url = f"https://accounts.gcp.databricks.com/api/2.0/accounts/{ACCOUNT_ID}/workspaces"
workspaces = requests.get(
    url,
    headers={"Authorization": f"Bearer {TOKEN}"}
).json()

rows = [
    (
        str(w["workspace_id"]),
        w.get("workspace_name"),
        w.get("cloud_resource_container", {})
         .get("gcp", {})
         .get("project_id")
    )
    for w in workspaces
]
df = spark.createDataFrame(
    rows,
    ["workspace_id", "workspace_name", "project_id"]
)

Then persist in a mapping table like workspace_project_map and enrich billing as:

SELECT
  u.*,
  m.project_id
FROM system.billing.usage u
LEFT JOIN finops.workspace_project_map m
  ON u.workspace_id = m.workspace_id

At present, system.access.workspaces_latest provides workspace metadata such as workspace_id, workspace_name, workspace_url, and status but doesn't expose the GCP project_id as mentioned in docs too.

So the practical approach for now could be API → mapping table → billing enrichment, while also watching for Databricks to add project_id directly to a system table in the future.