- 282 Views
- 2 replies
- 1 kudos
Building a Large-Scale Supply Chain Simulation Platform on Databricks
A Data & AI–Driven Decision Engine for Modern Retail NetworksIntroductionIn modern retail, supply chains are no longer static networks — they are living, adaptive systems that must continuously respond to customer demand, fulfillment speed expectatio...
- 282 Views
- 2 replies
- 1 kudos
- 1 kudos
Great article! would love to know more as I have a very similar case with a concrete customer. Thanks, S.
- 1 kudos
- 28 Views
- 0 replies
- 1 kudos
It's time to treat AI as a peer, not a tool. What if your AI already knew Databricks?
We need to stop treating AI as a tool. It's time to treat it as a peer.I've been building a library of reusable skills for Claude — structured instructions that let AI agents handle complex, repetitive development workflows on Databricks and Azure AI...
- 28 Views
- 0 replies
- 1 kudos
- 57 Views
- 0 replies
- 0 kudos
I Tried Building an Agentic AI System for Construction. Here’s What Actually Worked.
Most construction teams don’t really have a data problem, at least not in the way we usually think about it. They already have dashboards everywhere. Finance has reports, project managers have schedule views, field teams have inspection logs. Everyon...
- 57 Views
- 0 replies
- 0 kudos
- 183 Views
- 0 replies
- 1 kudos
Databricks Transactions - Enforcing Business Validation Rules via SQL SIGNAL
Combining SIGNAL statement with ATOMIC transactions in Databricks saves us from managing commits & rollbacks along with managing custom validations seamlessly - something that modern big data ETL frameworks struggle to deliver cleanly. They give the ...
- 183 Views
- 0 replies
- 1 kudos
- 284 Views
- 1 replies
- 3 kudos
Secure Credit Card Partner Enablement Using Databricks Clean Rooms
How Digital Payment Lending Platforms Can Collaborate with Banks Without Exposing Sensitive Data1. Business Context & Regulatory RealityIn 2020, large Indian fintech platforms faced a unique regulatory constraint: NBFC‑led digital platforms were not ...
- 284 Views
- 1 replies
- 3 kudos
- 3 kudos
This is a solid breakdown of how secure data collaboration can be done without exposing sensitive information. The Clean Room approach really stands out because it shifts the model from data sharing to controlled computation, which is exactly what re...
- 3 kudos
- 447 Views
- 2 replies
- 7 kudos
Databricks Multi-Table Transactions - Part 1
If you've ever worked on an insurance data warehouse, or really any warehouse where data arrives from different systems at different times, you know the pain of keeping things in sync. I spent years building data warehouses for a property and casual...
- 447 Views
- 2 replies
- 7 kudos
- 209 Views
- 0 replies
- 1 kudos
𝗦𝗜𝗘𝗠 𝗶𝘀 𝗹𝗲𝗴𝗮𝗰𝘆. Here's why, and what becomes possible when you move security operations
I've spent years migrating SOC operations from traditional SIEM to Databricks. Not because it's trendy, but because SIEM has fundamental problems that no vendor update will fix: proprietary query languages that lock you in, no version control or test...
- 209 Views
- 0 replies
- 1 kudos
- 475 Views
- 9 replies
- 6 kudos
Streaming Failure Models: Why "It Didn't Crash" Is the Worst Outcome
Most Databricks streaming failures don't look dramatic.No cluster termination. No red wall of errors. The UI says RUNNING — and your customers start reporting nonsense.I wrote about the incident that changed how we think about streaming jobs on share...
- 475 Views
- 9 replies
- 6 kudos
- 6 kudos
Completely agree, production war stories are worth more than any documentation. I’ve eaten enough teeth on production data lake issues to write my own chapter on what can go wrong, whether that’s deploying Databricks in financial institutions or bein...
- 6 kudos
- 127 Views
- 0 replies
- 1 kudos
Built a NiFi processor for Zerobus Ingest - gotchas the docs won’t tell you
Zerobus went GA on February 23rd. Connector ecosystem: empty. I run NiFi for security telemetry so I built the processor myself. Apache 2.0, source on GitHub.NiFi uses NAR packaging — each archive gets its own classloader. The Zerobus Java SDK is JNI...
- 127 Views
- 0 replies
- 1 kudos
- 350 Views
- 0 replies
- 2 kudos
Databricks Multi Table Transactions - All Data or Nothing
Databricks introduces multi-table transactions, allowing operations across multiple Delta tables to execute as a single atomic unit. Delta Lake has provided ACID guarantees at the table level, but ensuring atomicity across multiple tables previously ...
- 350 Views
- 0 replies
- 2 kudos
- 258 Views
- 1 replies
- 2 kudos
Multi-Task on a Shared Cluster — Why That's Also Not Enough
Part 2 of 3 — Databricks Streaming ArchitectureThe instinct after Part 1 was obvious.If running eight queries in one task means one failure can hide while others keep running — split them into multiple tasks. Separate concerns. Give each component it...
- 258 Views
- 1 replies
- 2 kudos
- 2 kudos
Part 1: Streaming Failure Models: Why "It Didn't Crash" Is the Worst OutcomePart 3: One Cluster per Task — Proven, Ready, and Waiting
- 2 kudos
- 207 Views
- 0 replies
- 1 kudos
Enterprise Data Platform Architecture on Azure with Databricks
Hi everyone,I recently wrote an article on designing an enterprise-scale data platform architecture using Azure and Databricks.The article covers:• End-to-end architecture for enterprise data platforms• Data ingestion using Azure Data Factory and Kaf...
- 207 Views
- 0 replies
- 1 kudos
- 229 Views
- 0 replies
- 4 kudos
One Policy to Mask Them All: ABAC + VARIANT in Unity Catalog
Databricks ABAC lets you apply a single schema-level policy across columns of any data type — no more managing one mask function per type. Here's how to use the VARIANT data type to make it work. If you've implemented column masking in Unity Catalog,...
- 229 Views
- 0 replies
- 4 kudos
- 155 Views
- 0 replies
- 1 kudos
One Cluster per Task — Proven, Ready, and Waiting
Part 3 of 3: Databricks Streaming ArchitectureBy the end of Part 1 & Part 2, we knew what the real answer was. We just hadn’t committed to it yet.Not because it wouldn’t work. We tested it. We documented it. The code was ready. The answer was one clu...
- 155 Views
- 0 replies
- 1 kudos
- 225 Views
- 0 replies
- 2 kudos
Building a Hybrid Lakehouse: Strategic Use of Apache Hudi and Delta Lake in Databricks
Apache Hudi and Delta Lake are built for different workloads. Hudi is optimised for high-frequency writes; Delta Lake is built for fast, reliable reads. Using one format across the entire data platform forces an unnecessary trade-off high ingestion c...
- 225 Views
- 0 replies
- 2 kudos
| User | Count |
|---|---|
| 87 | |
| 71 | |
| 44 | |
| 41 | |
| 41 |
