- 3563 Views
- 1 replies
- 1 kudos
When Did the Data Go Wrong? Using Delta Lake Time Travel for Investigation in Databricks
I. IntroductionData pipelines are the lifeblood of modern data-driven organizations. However, even the most robust pipelines can experience unexpected issues: data corruption, erroneous updates, or sudden data drops. When these problems occur, quickl...
- 3563 Views
- 1 replies
- 1 kudos
- 1 kudos
Great thought to use delta time travel to determine when data drift starts!But this only works as long as retention policies allow it. With vacuum and stricter runtime enforcement in newer dbx versions, older snapshots may not be there when you need ...
- 1 kudos
- 870 Views
- 0 replies
- 1 kudos
Intelligence Studio -- Open-Source AI-Powered API Explorer for Databricks
Hi everyone,I built Intelligence Studio, an open-source workbench that lets you browse, test, analyse, and integrate with 640+ Databricks REST APIs -- all from one interface. No more juggling docs, curl commands, Postman collections, and multiple bro...
- 870 Views
- 0 replies
- 1 kudos
- 1757 Views
- 7 replies
- 6 kudos
What's Your Biggest AI Pet Peeve?
Hey Team, in my last post I asked how much AI has actually changed your day to day, and the responses were fantastic. But let's talk about the other side for a minute. I'll go first — I've started second-guessing almost everything I see on social med...
- 1757 Views
- 7 replies
- 6 kudos
- 6 kudos
AI tools generate code & pipelines that work functionally but ignore efficiency, scalability & cloud implications. I have bumped into belowCode generated by AI does a SQL cross join as its simple for most natural language queries that works but kills...
- 6 kudos
- 1175 Views
- 0 replies
- 2 kudos
Databricks Multi-Table Transactions - Part 2
In Part 1, we covered why multi-table transactions matter. Now let's build one. We'll create the tables from the claim wrap-up scenario, load sample P&C insurance data, and walk through what happens when the wrap-up succeeds, when it fails, and when...
- 1175 Views
- 0 replies
- 2 kudos
- 2657 Views
- 1 replies
- 5 kudos
Solving Multi-Dimension Analytics in Databricks Dashboards with Views and Metric Views
If you've ever built a dashboard where you needed to track the same data across two different date dimensions, you know the frustration. You get the first chart working. You add the second. Then you realise cross-filtering just stopped working. I re...
- 2657 Views
- 1 replies
- 5 kudos
- 5 kudos
Thanks for sharing great example with detailed explanation.
- 5 kudos
- 3525 Views
- 0 replies
- 1 kudos
How to actually get job_id and run_id in a Databricks Python wheel task (Avoid Hallucinations)
We needed job_id and run_id in a custom metrics Delta table so we could join to `system.lakeflow.job_run_timeline`. Tried four approaches before finding the one that works on serverless compute.What doesn't workspark.conf.get("spark.databricks.job.id...
- 3525 Views
- 0 replies
- 1 kudos
- 1950 Views
- 3 replies
- 1 kudos
PostgreSQL to Databricks made simpler with Lakeflow Connect(Public Preview)
PostgreSQL to Databricks made simpler with Lakeflow Connect (Public Preview).Databricks has introduced a PostgreSQL connector in Lakeflow Connect (Public Preview), enabling ingestion of PostgreSQL data into the Lakehouse using logical replication.Ins...
- 1950 Views
- 3 replies
- 1 kudos
- 1 kudos
You can enable or disable previews using account console for account level previews and workspace for workspace level previews.More details here
- 1 kudos
- 1137 Views
- 0 replies
- 2 kudos
Turning Lakehouse into Brainhouse via Knowledge Graphs
Organizations solved the challenge of collecting, cleaning & governing structured data at scale via Delta Lake and Unity Catalog in Lakehouse. You have world class lineage, permissions, RBAC, ABAC and schemas as the nervous system. The nervous system...
- 1137 Views
- 0 replies
- 2 kudos
- 1685 Views
- 2 replies
- 1 kudos
Building a Large-Scale Supply Chain Simulation Platform on Databricks
A Data & AI–Driven Decision Engine for Modern Retail NetworksIntroductionIn modern retail, supply chains are no longer static networks — they are living, adaptive systems that must continuously respond to customer demand, fulfillment speed expectatio...
- 1685 Views
- 2 replies
- 1 kudos
- 1 kudos
Great article! would love to know more as I have a very similar case with a concrete customer. Thanks, S.
- 1 kudos
- 1722 Views
- 0 replies
- 1 kudos
It's time to treat AI as a peer, not a tool. What if your AI already knew Databricks?
We need to stop treating AI as a tool. It's time to treat it as a peer.I've been building a library of reusable skills for Claude — structured instructions that let AI agents handle complex, repetitive development workflows on Databricks and Azure AI...
- 1722 Views
- 0 replies
- 1 kudos
- 2117 Views
- 0 replies
- 1 kudos
I Tried Building an Agentic AI System for Construction. Here’s What Actually Worked.
Most construction teams don’t really have a data problem, at least not in the way we usually think about it. They already have dashboards everywhere. Finance has reports, project managers have schedule views, field teams have inspection logs. Everyon...
- 2117 Views
- 0 replies
- 1 kudos
- 853 Views
- 0 replies
- 1 kudos
Databricks Transactions - Enforcing Business Validation Rules via SQL SIGNAL
Combining SIGNAL statement with ATOMIC transactions in Databricks saves us from managing commits & rollbacks along with managing custom validations seamlessly - something that modern big data ETL frameworks struggle to deliver cleanly. They give the ...
- 853 Views
- 0 replies
- 1 kudos
- 1635 Views
- 0 replies
- 1 kudos
𝗦𝗜𝗘𝗠 𝗶𝘀 𝗹𝗲𝗴𝗮𝗰𝘆. Here's why, and what becomes possible when you move security operations
I've spent years migrating SOC operations from traditional SIEM to Databricks. Not because it's trendy, but because SIEM has fundamental problems that no vendor update will fix: proprietary query languages that lock you in, no version control or test...
- 1635 Views
- 0 replies
- 1 kudos
- 2244 Views
- 9 replies
- 6 kudos
Streaming Failure Models: Why "It Didn't Crash" Is the Worst Outcome
Most Databricks streaming failures don't look dramatic.No cluster termination. No red wall of errors. The UI says RUNNING — and your customers start reporting nonsense.I wrote about the incident that changed how we think about streaming jobs on share...
- 2244 Views
- 9 replies
- 6 kudos
- 6 kudos
Completely agree, production war stories are worth more than any documentation. I’ve eaten enough teeth on production data lake issues to write my own chapter on what can go wrong, whether that’s deploying Databricks in financial institutions or bein...
- 6 kudos
- 516 Views
- 0 replies
- 1 kudos
Built a NiFi processor for Zerobus Ingest - gotchas the docs won’t tell you
Zerobus went GA on February 23rd. Connector ecosystem: empty. I run NiFi for security telemetry so I built the processor myself. Apache 2.0, source on GitHub.NiFi uses NAR packaging — each archive gets its own classloader. The Zerobus Java SDK is JNI...
- 516 Views
- 0 replies
- 1 kudos
| User | Count |
|---|---|
| 85 | |
| 76 | |
| 73 | |
| 60 | |
| 43 |
