I've spent years migrating SOC operations from traditional SIEM to Databricks. Not because it's trendy, but because SIEM has fundamental problems that no vendor update will fix: proprietary query languages that lock you in, no version control or testing for detection rules, retention costs that force teams to drop data, and hunting workflows that mean fighting the tool instead of finding threats.
On a Data Lake, all of that changes. Your telemetry lives in Delta tables. Your detection logic is SQL or Python, versioned in Git, tested against historical data. Retention is cheap storage, not expensive licensing. And hunting becomes: describe what you're looking for, query across all sources, pivot in one view.
But here's what I think is underexplored: ๐๐ต๐ฒ ๐ฐ๐ผ๐บ๐ฏ๐ถ๐ป๐ฎ๐๐ถ๐ผ๐ป ๐ผ๐ณ ๐๐ผ๐๐ป๐ฑ๐ฎ๐๐ถ๐ผ๐ป ๐ ๐ผ๐ฑ๐ฒ๐น๐ ๐ฎ๐ป๐ฑ ๐๐ฎ๐๐ฎ๐ฏ๐ฟ๐ถ๐ฐ๐ธ๐ ๐๐ฝ๐ฝ๐ ๐ผ๐ฝ๐ฒ๐ป๐ ๐ฎ ๐ฐ๐ผ๐บ๐ฝ๐น๐ฒ๐๐ฒ๐น๐ ๐ป๐ฒ๐ ๐ฐ๐ฎ๐๐ฒ๐ด๐ผ๐ฟ๐ ๐ผ๐ณ ๐๐ฒ๐ฐ๐๐ฟ๐ถ๐๐ ๐๐ผ๐ผ๐น๐ถ๐ป๐ด that didn't exist before.
Think about it:
โ A SOC analyst can describe a hypothesis in plain English and get SQL generated from the actual table schema via DESCRIBE TABLE. No proprietary query language to learn.
โ Foundation Model endpoints are available pay-per-token, no deployment, no GPU management. Any Databricks App can call them.
โ Databricks Apps let you ship a full internal tool in three files: app.py, app.yaml, requirements.txt. Streamlit frontend, SDK auth, serverless compute. No infrastructure.
โ Delta tables with MERGE INTO and time travel give you audit-grade persistence for free. Every investigation, every status change, every hunt is versioned and queryable.
โ Unity Catalog handles governance. The app doesn't manage permissions, the platform does.
This means ๐ฎ๐ป๐ ๐๐ฒ๐ฐ๐๐ฟ๐ถ๐๐ ๐๐ฒ๐ฎ๐บ ๐ฐ๐ฎ๐ป ๐ฏ๐๐ถ๐น๐ฑ ๐ฒ๐
๐ฎ๐ฐ๐๐น๐ ๐๐ต๐ฎ๐ ๐๐ต๐ฒ๐ ๐ป๐ฒ๐ฒ๐ฑ. Not what a vendor decided to ship. Not a one-size-fits-all dashboard. Custom tooling, built by the people who know the environment best, deployed in minutes.
To test this idea I built a small PoC: a threat hunting app running on Databricks Apps. Three views: hypothesis-to-SQL workspace, entity timeline across all sources, and a kanban hunt board backed by Delta. One Python file. Took about an hour.
I'm not a frontend developer. I'm not even a threat hunter. I'm a security architect who wanted to see how far the platform goes. The answer: further than I expected.
๐ง๐ต๐ฒ ๐ถ๐ป๐๐ฒ๐ฟ๐ฒ๐๐๐ถ๐ป๐ด ๐พ๐๐ฒ๐๐๐ถ๐ผ๐ป ๐ถ๐๐ป'๐ ๐๐ต๐ฎ๐ ๐ ๐ฏ๐๐ถ๐น๐. It's what happens when real SOC teams, detection engineers, and threat hunters start building their own tools on Databricks Apps with Foundation Models. The platform is ready. The ecosystem of security-specific apps is not. Yet.
Full walkthrough with architecture and code:
https://dere.la/posts/siem-legacy-threathunt/
Source code (Apache 2.0):
https://github.com/us3r/databricks-threathunt
