- 729 Views
- 9 replies
- 6 kudos
Streaming Failure Models: Why "It Didn't Crash" Is the Worst Outcome
Most Databricks streaming failures don't look dramatic.No cluster termination. No red wall of errors. The UI says RUNNING — and your customers start reporting nonsense.I wrote about the incident that changed how we think about streaming jobs on share...
- 729 Views
- 9 replies
- 6 kudos
- 6 kudos
Completely agree, production war stories are worth more than any documentation. I’ve eaten enough teeth on production data lake issues to write my own chapter on what can go wrong, whether that’s deploying Databricks in financial institutions or bein...
- 6 kudos
- 219 Views
- 0 replies
- 0 kudos
🚀 LDP Tax Pipeline — Spark Declarative Pipelines on macOS (Without Databricks)
LDP Tax Pipeline — Spark Declarative Pipelines on macOS (Without Databricks)Excited to share my latest hands-on implementation of a LakeFlow Declarative Pipeline (LDP) built locally using Apache Spark 4.1 Declarative Pipelines — running entirely on ...
- 219 Views
- 0 replies
- 0 kudos
- 460 Views
- 0 replies
- 1 kudos
SQL Scripting in Apache Sparkâ„¢ 4.0
The Apache Sparkâ„¢ 4.0 introduces a new feature for SQL developers and data engineers: SQL Scripting. As such, this feature enhances the power and extends the flexibility of Spark SQL, enabling users to write procedural code within SQL queries, with t...
- 460 Views
- 0 replies
- 1 kudos