Databricks Community

data_architect2 · Tuesday

Hello everyone,

We are planning to roll out Genie Spaces as a BI Agent and Genie Code as a Pair Programmer for multiple user groups. Our current Genie guardrails are set by inheriting workspace policies and Unity Catalog permissions with ABAC. Additionally, Genie data access is logged into audit tables, and it has built-in safety mechanisms to control output.

We are looking to set up a robust, periodic OWASP LLM vulnerability risk audit on our Genie environment. If anyone has pointers, best practices, or documentation on how to handle this kind of exercise, your guidance would be greatly appreciated.

Thank you

GabFernandes · yesterday

Hi @data_architect2 ,

Great question — it's good to see organizations proactively thinking about LLM security audits for their AI assistants. Your current setup (UC permissions + ABAC + audit logging) already covers several OWASP LLM Top 10 risks out-of-the-box. Here's a structured approach for a periodic audit:

1. Start with the Databricks AI Security Framework (DASF)

Databricks published the DASF whitepaper which maps real-world AI/ML attack scenarios to mitigations. It's directly aligned with OWASP and NIST AI RMF. Use it as your risk register baseline.

2. OWASP LLM Top 10 — Mapping to Genie's Built-In Controls

OWASP LLM Risk	Genie Built-In Mitigation	Your Audit Action
LLM01: Prompt Injection	Genie only generates SQL against curated tables/instructions — no arbitrary code execution in Spaces. Genie Code runs in sandboxed compute with UC permissions enforced.	Periodically review system.access.audit (service aibiGenie) and system.access.assistant_events for anomalous prompts or unusual query patterns.
LLM02: Insecure Output Handling	Genie Spaces returns only SQL results (no rendered HTML/scripts). Genie Code output is sandboxed in notebooks.	Validate that downstream consumers don't blindly trust AI-generated output without human review (e.g., "Request for Review" workflow in Spaces).
LLM03: Training Data Poisoning	Databricks does NOT train models on your data or prompts. Zero data retention policy with model partners.	Document this in your risk register as a mitigated/transferred risk.
LLM04: Model Denial of Service	Document this in your risk register as a mitigated/transferred risk.	Monitor warehouse utilization via system.billing for unexpected spikes tied to Genie usage.
LLM05: Supply Chain Vulnerabilities	Models are managed by Databricks (Azure OpenAI / Anthropic endpoints with TLS). No user-supplied model weights.	Confirm compliance certifications (SOC2, ISO 27001, TISAX) cover Genie Spaces (they do — see compliance docs).
LLM06: Sensitive Information Disclosure	Your strongest control: UC permissions + ABAC + row filters + column masks are enforced per-user at query time in Genie.	Audit: run SHOW ROW FILTERS ON <table> and SHOW COLUMN MASKS ON <table> periodically to confirm policies are still applied. Cross-reference with system.access.audit to ensure no privilege escalation.
LLM07: Insecure Plugin Design	Genie Spaces: only trusted assets (curated SQL queries + UC functions) execute. Genie Code: sandboxed to user's own UC permissions.	Audit the trusted queries/functions list in each Genie Space periodically — ensure no overly-permissive functions were added.
LLM08: Excessive Agency	Genie Spaces can ONLY read data (SELECT). No DDL/DML. Genie Code actions are bound by UC permissions of the individual user.	Confirm via audit logs that no admin-level users are using Genie Spaces with overly broad permissions.
LLM09: Overreliance	N/A (organizational risk)	Train users to validate AI answers. Leverage the built-in thumbs-up/down feedback + "Request for Review" workflow in Genie Spaces as a quality signal.
LLM10: Model Theft	Models are hosted by Databricks/partners — not accessible to customers.	Not applicable in this architecture.

3. Practical Audit Queries (run periodically)

-- All Genie Space interactions in last 30 days (who asked what, which space)
SELECT
  event_date,
  user_identity.email,
  action_name,
  request_params.space_id,
  request_params
FROM system.access.audit
WHERE service_name = 'aibiGenie'
  AND event_date >= current_date() - INTERVAL 30 DAYS
ORDER BY event_date DESC;

-- Genie Code (pair programmer) usage patterns
SELECT *
FROM system.access.assistant_events
WHERE event_date >= current_date() - INTERVAL 30 DAYS;

-- Detect unusual data access patterns triggered by Genie
SELECT
  user_identity.email,
  request_params.commandText,
  action_name
FROM system.access.audit
WHERE service_name = 'databrickssql'
  AND source_ip_address IS NOT NULL
  AND event_date >= current_date() - INTERVAL 7 DAYS
  AND request_params.commandText LIKE '%SELECT%'
ORDER BY event_time DESC;

4. Recommended Audit Cadence

Frequency	Activity
Weekly	Review negative feedback (thumbs-down) in Genie Spaces for potential misuse or data leakage attempts
Monthly	Audit new trusted assets (queries/functions) added to Genie Spaces; review UC permission changes
Quarterly	Full OWASP LLM Top 10 checklist review against current Genie config; DASF risk register update
Annually	Penetration test / red-team exercise (prompt injection attempts against your Genie Spaces with test users)

5. Key Documentation Links

TL;DR: Genie's architecture already mitigates most OWASP LLM risks by design (no training on your data, UC-enforced access, SQL-only output, sandboxed execution). Your periodic audit should focus on: (1) monitoring audit logs for anomalous prompts/patterns, (2) validating UC policies (ABAC/row filters) are correctly applied, (3) reviewing trusted assets in Genie Spaces, and (4) an annual red-team exercise for prompt injection.

If my answer was helpful, please consider marking it as accepted solution!

View solution in original post

GabFernandes · yesterday