The Weekly Digest • May 04 – May 10
Community Pulse
This week's top threads, breakthroughs, and knowledge drops from the community. 🗞💬
🏅 Top Contributors This Week
Contributors leading with curiosity and kindness:
+ Top Brickster contributor:
Note: Names are listed in no particular order.
📖 Knowledge Hub
📃 MVP Articles
- Watermark-Based Incremental Ingestion (Lakeflow Connect query-based capture)
- How Switching from JDBC/ODBC Clusters to Serverless SQL Warehouses Boosted Our Power BI Performance
- Disable Tasks in Databricks Lakeflow Jobs: A Powerful Feature for Flexible Workflow Orchestration
- Ingestion without CDF
- Why You Cannot Choose the SQL Warehouse in Databricks Chat & Assistant Features?
- Lakehouse Sync
✍️ Community Articles
- Databricks Lake base - Enterprise Healthcare Data Intelligence via Bidirectional Data Sync
- Databricks Lakebase Just Eliminated the Wall Between Applications and Analytics.
- Finally! Databricks lets you disable tasks without hacks
- Managed vs External Tables in Unity Catalog: The Decision That’s Silently Inflating Your Cloud Bill
- How Databricks Genie Turns Collaboration Tools into AI-Powered Intelligence Platforms
- Scaling Enterprise AI with Databricks Without Losing Control
- Databricks Kafka Multi-Stream Ingestion Architecture: Scaling Beyond Single-Stream Bottlenecks
📌 Technical Blogs
Deep dives from the Databricks engineering team and partners 🔬:
- Performing DML Operations on Unity Catalog Managed Tables from External Engines
- Semantic Caching for LLM Applications with Databricks Lakebase and pgvector
- Delta Lake Under the Hood: What Every Data Engineer Should Know
- Mastering Delta Lake MERGE Performance: Why It Slows Down and How to Fix It
- Beyond RAG: Databricks' Unified Approach to Document Insights
- Three MCPs, One Answer: Building a Data Quality Monitor on Databricks Apps
- Deploying Django apps on Databricks Apps with Lakebase
- [PARTNER BLOG] Revolutionizing Enterprise Data with Databricks Agent Bricks: Tale of Two Industries
The most heavily trafficked threads of the week 👀:
- DAB git - sometimes doesn't see modules
- Lakebridge reconciliation code keeps running continuously without Spark jobs or errors
- Compute tab doesn't show and doesn't give the option to create a cluster
- Databricks Workflow for Sharing Delta Table Data via Email (Text & Attachment)
- Unity Catalog - How to read prod data in dev with appropriate read-only access?
Learning Series
🎯 From Blocked to Breakthrough
Here are the top solutions locked in this week:
- run_if condition to handle prior task excluded?
- AI Playground access Tools(SQL Function) with spark connect PERMISSION_DENIED
- Need access to "Databricks Workspace Walkthrough" Folder
- Lakeflow Declarative Pipeline queue
- How to limit max concurrent tasks runs in a job?
- +.....
Thanks for being here. 🙏
This digest exists because of the members who show up, contribute, and lift each other up.
See you in the threads!
