System tables for vector search index?

Rjdudley — Mon, 10 Nov 2025 17:24:16 GMT

Are there system tables or log files for monitoring vector search index searches? I didn't see vector search mentioned on the system tables documentation (Monitor account activity with system tables - Azure Databricks | Microsoft Learn).

Re: System tables for vector search index?

mark_ott — Tue, 11 Nov 2025 10:33:40 GMT

Yes, there are system tables and log files specifically for monitoring vector search index activity in Azure Databricks, though they aren't referenced directly in the main "system tables" documentation you linked.

Vector Search Monitoring via System Tables

The primary system table for monitoring usage and costs associated with vector search indexes and endpoints is the system.billing.usage table. This table tracks billable operations, including those related to vector search workloads like ingest, storage, and serving.
Example queries are provided in the Mosaic AI Vector Search documentation to isolate vector search workload consumption by endpoint, ingest, and storage activity.

Dedicated Log Files and Audit Tracking

Azure Monitor includes a specific log table named DatabricksVectorSearch, which is designed for audit logging of vector search operations. This log captures essential metadata such as action names, request IDs, user identities, response data, session information, timestamps, and client IP addresses, providing a detailed operational and security audit trail.
This table includes columns like ActionName, OperationName, RequestParams, UserAgent, TimeGenerated, among others—information valuable for tracking individual vector search queries and user activity.

Additional Notes

The billable usage table and the DatabricksVectorSearch table are your best sources for monitoring and auditing both general usage and specific search activities for vector indexes in Databricks.
These audit and usage logs are accessible within Azure Monitor and Log Analytics Workspaces, not only from inside Databricks system tables.
No single table aggregates traditional SQL query-level detail for vector search (such as exact query text), but action-level audit logs provide broad monitoring coverage.

For the latest setup instructions and columns list, refer to:

Re: System tables for vector search index?

Rjdudley — Tue, 11 Nov 2025 14:58:14 GMT

Thanks @mark_ott are there plans to make the vector search a "proper" system table in Databricks rather than using an Azure Monitor table?

topic System tables for vector search index? in Generative AI

System tables for vector search index?

Re: System tables for vector search index?

Vector Search Monitoring via System Tables

Dedicated Log Files and Audit Tracking

Additional Notes

Re: System tables for vector search index?