Re: Genie LLM audit

GabFernandes · yesterday

Great question — it's good to see organizations proactively thinking about LLM security audits for their AI assistants. Your current setup (UC permissions + ABAC + audit logging) already covers several OWASP LLM Top 10 risks out-of-the-box. Here's a structured approach for a periodic audit:

1. Start with the Databricks AI Security Framework (DASF)

Databricks published the DASF whitepaper which maps real-world AI/ML attack scenarios to mitigations. It's directly aligned with OWASP and NIST AI RMF. Use it as your risk register baseline.

2. OWASP LLM Top 10 — Mapping to Genie's Built-In Controls

OWASP LLM Risk	Genie Built-In Mitigation	Your Audit Action
LLM01: Prompt Injection	Genie only generates SQL against curated tables/instructions — no arbitrary code execution in Spaces. Genie Code runs in sandboxed compute with UC permissions enforced.	Periodically review system.access.audit (service aibiGenie) and system.access.assistant_events for anomalous prompts or unusual query patterns.
LLM02: Insecure Output Handling	Genie Spaces returns only SQL results (no rendered HTML/scripts). Genie Code output is sandboxed in notebooks.	Validate that downstream consumers don't blindly trust AI-generated output without human review (e.g., "Request for Review" workflow in Spaces).
LLM03: Training Data Poisoning	Databricks does NOT train models on your data or prompts. Zero data retention policy with model partners.	Document this in your risk register as a mitigated/transferred risk.
LLM04: Model Denial of Service	Document this in your risk register as a mitigated/transferred risk.	Monitor warehouse utilization via system.billing for unexpected spikes tied to Genie usage.
LLM05: Supply Chain Vulnerabilities	Models are managed by Databricks (Azure OpenAI / Anthropic endpoints with TLS). No user-supplied model weights.	Confirm compliance certifications (SOC2, ISO 27001, TISAX) cover Genie Spaces (they do — see compliance docs).
LLM06: Sensitive Information Disclosure	Your strongest control: UC permissions + ABAC + row filters + column masks are enforced per-user at query time in Genie.	Audit: run SHOW ROW FILTERS ON <table> and SHOW COLUMN MASKS ON <table> periodically to confirm policies are still applied. Cross-reference with system.access.audit to ensure no privilege escalation.
LLM07: Insecure Plugin Design	Genie Spaces: only trusted assets (curated SQL queries + UC functions) execute. Genie Code: sandboxed to user's own UC permissions.	Audit the trusted queries/functions list in each Genie Space periodically — ensure no overly-permissive functions were added.
LLM08: Excessive Agency	Genie Spaces can ONLY read data (SELECT). No DDL/DML. Genie Code actions are bound by UC permissions of the individual user.	Confirm via audit logs that no admin-level users are using Genie Spaces with overly broad permissions.
LLM09: Overreliance	N/A (organizational risk)	Train users to validate AI answers. Leverage the built-in thumbs-up/down feedback + "Request for Review" workflow in Genie Spaces as a quality signal.
LLM10: Model Theft	Models are hosted by Databricks/partners — not accessible to customers.	Not applicable in this architecture.

3. Practical Audit Queries (run periodically)

-- All Genie Space interactions in last 30 days (who asked what, which space)
SELECT
  event_date,
  user_identity.email,
  action_name,
  request_params.space_id,
  request_params
FROM system.access.audit
WHERE service_name = 'aibiGenie'
  AND event_date >= current_date() - INTERVAL 30 DAYS
ORDER BY event_date DESC;

-- Genie Code (pair programmer) usage patterns
SELECT *
FROM system.access.assistant_events
WHERE event_date >= current_date() - INTERVAL 30 DAYS;

-- Detect unusual data access patterns triggered by Genie
SELECT
  user_identity.email,
  request_params.commandText,
  action_name
FROM system.access.audit
WHERE service_name = 'databrickssql'
  AND source_ip_address IS NOT NULL
  AND event_date >= current_date() - INTERVAL 7 DAYS
  AND request_params.commandText LIKE '%SELECT%'
ORDER BY event_time DESC;

4. Recommended Audit Cadence

Frequency	Activity
Weekly	Review negative feedback (thumbs-down) in Genie Spaces for potential misuse or data leakage attempts
Monthly	Audit new trusted assets (queries/functions) added to Genie Spaces; review UC permission changes
Quarterly	Full OWASP LLM Top 10 checklist review against current Genie config; DASF risk register update
Annually	Penetration test / red-team exercise (prompt injection attempts against your Genie Spaces with test users)

5. Key Documentation Links

TL;DR: Genie's architecture already mitigates most OWASP LLM risks by design (no training on your data, UC-enforced access, SQL-only output, sandboxed execution). Your periodic audit should focus on: (1) monitoring audit logs for anomalous prompts/patterns, (2) validating UC policies (ABAC/row filters) are correctly applied, (3) reviewing trusted assets in Genie Spaces, and (4) an annual red-team exercise for prompt injection.

If my answer was helpful, please consider marking it as accepted solution!

View solution in original post