Before dropping a column, check whether Genie has been using it

[ivanvyd]

I was reviewing a schema change recently and wanted a quick answer to a simple question:

Could this column be used by a Genie Agent somewhere without me realizing it?

There's now a pretty useful way to check I want to share with you.

Databricks' March 25, 2026 release notes introduced entity_metadata.genie_space_id in both system.access.column_lineage and system.access.table_lineage. The field identifies the Genie Agent when a lineage record originates from it querying data (see March 2026 | Databricks on AWS).

For a column such as customer_segment, I can start with this query:

SELECT
    workspace_id,
    entity_metadata.genie_space_id AS genie_space_id,
    direct_access,
    COUNT(DISTINCT event_id) AS observed_events,
    MAX(event_time) AS last_seen
FROM system.access.column_lineage
WHERE source_table_full_name = 'prod.sales.orders'
  AND source_column_name = 'customer_segment'
  AND entity_metadata.genie_space_id IS NOT NULL
  AND event_date >= date_sub(current_date(), 90)
GROUP BY
    workspace_id,
    entity_metadata.genie_space_id,
    direct_access
ORDER BY last_seen DESC;

I can run this from a Unity Catalog-enabled workspace where I have access to the lineage system tables. 

I use the result set to inspect observed usage by workspace, Genie ID, and direct-access status. observed_events counts distinct lineage events. I do not treat it as a count of user questions or conversations. last_seen gives me the latest recorded lineage timestamp inside the selected window.

I keep indirect access in the result

direct_access = false means the source is a dependency discovered through view expansion.

A Genie Agent can query a view while the recorded lineage points to an underlying column. If I filter the query to direct access, I miss those records.

That distinction matters during a schema review because the team changing the base table may never see the base table referenced in the Genie-facing query.

An empty result does not prove the change is safe

I treat an empty result as absence of matching recorded lineage in the window I queried. I do not treat it as proof that nothing depends on the column.

A Genie Agent may not have used the column during the lookback period. Databricks emits lineage records when it can infer lineage, so the lineage tables do not capture every read or write event.

I also account for the scope and latency of the system tables. These lineage tables cover workspaces in the account within the same cloud region, and system-table updates are not real-time. Recent activity may not appear yet.

The SQL does not perform recursive dependency traversal through downstream tables that separate jobs populate.

A chain such as:

orders > ETL job > summary table > Genie

needs a broader dependency review. This query does not connect those separate lineage events for me.

I pair observed usage with configuration review

For higher-risk changes, I also inspect Genie configuration, including relevant agents that did not appear in the lineage results.

I check example SQL, instructions, join expressions, and referenced views or functions.

The Genie API can return a serialized configuration through:

GET /api/2.0/genie/spaces/{space_id}?include_serialized_space=true

Requesting that export requires at least CAN EDIT permission on the agent. The exported configuration includes data sources and instructions, including example SQL and function references.

I do not stop at the attached-table list. Genie can query other tables for which it has Unity Catalog permissions, including tables referenced in instructions or generated queries.

For table renames or drops, I use the same approach with system.access.table_lineage: I change the system table and remove the source_column_name predicate. The Genie metadata is available there as well.

My deployment review follows this sequence:

Proposed schema change > inspect Genie lineage > review dependencies and configurations > test > deploy

I use the lineage query as the starting point for investigating observed dependencies. It is evidence for the review, not automatic approval to change the schema.