I've spent years migrating SOC operations from traditional SIEM to Databricks. Not because it's trendy, but because SIEM has fundamental problems that no vendor update will fix: proprietary query languages that lock you in, no version control or testing for detection rules, retention costs that force teams to drop data, and hunting workflows that mean fighting the tool instead of finding threats.
On a Data Lake, all of that changes. Your telemetry lives in Delta tables. Your detection logic is SQL or Python, versioned in Git, tested against historical data. Retention is cheap storage, not expensive licensing. And hunting becomes: describe what you're looking for, query across all sources, pivot in one view.
But here's what I think is underexplored: 𝘁𝗵𝗲 𝗰𝗼𝗺𝗯𝗶𝗻𝗮𝘁𝗶𝗼𝗻 𝗼𝗳 𝗙𝗼𝘂𝗻𝗱𝗮𝘁𝗶𝗼𝗻 𝗠𝗼𝗱𝗲𝗹𝘀 𝗮𝗻𝗱 𝗗𝗮𝘁𝗮𝗯𝗿𝗶𝗰𝗸𝘀 𝗔𝗽𝗽𝘀 𝗼𝗽𝗲𝗻𝘀 𝗮 𝗰𝗼𝗺𝗽𝗹𝗲𝘁𝗲𝗹𝘆 𝗻𝗲𝘄 𝗰𝗮𝘁𝗲𝗴𝗼𝗿𝘆 𝗼𝗳 𝘀𝗲𝗰𝘂𝗿𝗶𝘁𝘆 𝘁𝗼𝗼𝗹𝗶𝗻𝗴 that didn't exist before.
Think about it:
→ A SOC analyst can describe a hypothesis in plain English and get SQL generated from the actual table schema via DESCRIBE TABLE. No proprietary query language to learn.
→ Foundation Model endpoints are available pay-per-token, no deployment, no GPU management. Any Databricks App can call them.
→ Databricks Apps let you ship a full internal tool in three files: app.py, app.yaml, requirements.txt. Streamlit frontend, SDK auth, serverless compute. No infrastructure.
→ Delta tables with MERGE INTO and time travel give you audit-grade persistence for free. Every investigation, every status change, every hunt is versioned and queryable.
→ Unity Catalog handles governance. The app doesn't manage permissions, the platform does.
This means 𝗮𝗻𝘆 𝘀𝗲𝗰𝘂𝗿𝗶𝘁𝘆 𝘁𝗲𝗮𝗺 𝗰𝗮𝗻 𝗯𝘂𝗶𝗹𝗱 𝗲𝘅𝗮𝗰𝘁𝗹𝘆 𝘄𝗵𝗮𝘁 𝘁𝗵𝗲𝘆 𝗻𝗲𝗲𝗱. Not what a vendor decided to ship. Not a one-size-fits-all dashboard. Custom tooling, built by the people who know the environment best, deployed in minutes.
To test this idea I built a small PoC: a threat hunting app running on Databricks Apps. Three views: hypothesis-to-SQL workspace, entity timeline across all sources, and a kanban hunt board backed by Delta. One Python file. Took about an hour.
I'm not a frontend developer. I'm not even a threat hunter. I'm a security architect who wanted to see how far the platform goes. The answer: further than I expected.
𝗧𝗵𝗲 𝗶𝗻𝘁𝗲𝗿𝗲𝘀𝘁𝗶𝗻𝗴 𝗾𝘂𝗲𝘀𝘁𝗶𝗼𝗻 𝗶𝘀𝗻'𝘁 𝘄𝗵𝗮𝘁 𝗜 𝗯𝘂𝗶𝗹𝘁. It's what happens when real SOC teams, detection engineers, and threat hunters start building their own tools on Databricks Apps with Foundation Models. The platform is ready. The ecosystem of security-specific apps is not. Yet.
Full walkthrough with architecture and code:
https://dere.la/posts/siem-legacy-threathunt/
Source code (Apache 2.0):
https://github.com/us3r/databricks-threathunt
