Yes, there are system tables and log files specifically for monitoring vector search index activity in Azure Databricks, though they aren't referenced directly in the main "system tables" documentation you linked.
Vector Search Monitoring via System Tables
-
The primary system table for monitoring usage and costs associated with vector search indexes and endpoints is the system.billing.usage table. This table tracks billable operations, including those related to vector search workloads like ingest, storage, and serving.โ
-
Example queries are provided in the Mosaic AI Vector Search documentation to isolate vector search workload consumption by endpoint, ingest, and storage activity.โ
Dedicated Log Files and Audit Tracking
-
Azure Monitor includes a specific log table named DatabricksVectorSearch, which is designed for audit logging of vector search operations. This log captures essential metadata such as action names, request IDs, user identities, response data, session information, timestamps, and client IP addresses, providing a detailed operational and security audit trail.โ
-
This table includes columns like ActionName, OperationName, RequestParams, UserAgent, TimeGenerated, among othersโinformation valuable for tracking individual vector search queries and user activity.โ
Additional Notes
-
The billable usage table and the DatabricksVectorSearch table are your best sources for monitoring and auditing both general usage and specific search activities for vector indexes in Databricks.
-
These audit and usage logs are accessible within Azure Monitor and Log Analytics Workspaces, not only from inside Databricks system tables.โ
-
No single table aggregates traditional SQL query-level detail for vector search (such as exact query text), but action-level audit logs provide broad monitoring coverage.
For the latest setup instructions and columns list, refer to:
