Hey Databricks Community! Join us for a free, community-sponsored BrickTalk on Thursday, October 29! BrickTalks is a community event series where Databricks experts share real-world use cases, demos, and practical insights for building with data and ...
- 297 Views
- 0 replies
- 0 kudos
Monday
DATABRICKS Asia-Pacific Learning Festival Free, virtual, hands-on Databricks training 21 & 22 October 2026 Virtual, labs included Free Ready to level up your Databricks skills? The Databricks Asia-Pacific Learning Festival is back, and ...
- 1883 Views
- 1 replies
- 1 kudos
2 weeks ago
FOR DATABRICKS LEARNERS Two free ways to grow yourDatabricks skills Both are free and live in the Databricks Community. Bookmark the one that fits where you are right now - or use both. 01 LEARNING PATHS Your Path to Certification What it is:...
- 1861 Views
- 3 replies
- 6 kudos
4 weeks ago
REGISTRATION IS NOW OPEN The Data + AI World Tour is heading your way A free, one-day event on how organizations are building AI they can trust to deliver real outcomes - by grounding models in their own enterprise data and business context. THE...
- 2505 Views
- 0 replies
- 3 kudos
09-01-2026
Free · Hands-on · Instructor-led AI/BI + Genie Launchpad From dashboards your team can trust to answers in plain language with Genie. A three-day arc of live, hands-on sessions, each built for a different role. Join only the day or days that fit you...
- 812 Views
- 0 replies
- 0 kudos
a week ago
Community Champion • September 2026 Meet Venugopal Dabbara Every month, our Community Champion Program recognizes a member who makes the Databricks Community better for everyone, through their knowledge, their willingness to help, and their commitmen...
- 639 Views
- 3 replies
- 6 kudos
2 weeks ago
Genome-wide association studies help identify genetic variations associated with a disease or trait. The Genome-Wide Association Studies Solution Accelerator builds on Glow, an open source project for genomic analysis, to provide a fast, scalable and...
- 161 Views
- 0 replies
- 1 kudos
Tuesday
Recap • September 14 – October 11 Community Pulse Four weeks of articles, learning content, and conversations – the highlights, all in one read. Community Stars The members who showed up and made a real difference : @ivanvyd @data_pulse @Tho...
- 189 Views
- 0 replies
- 3 kudos
Thursday
Community Activity
- 0 Views
- 0 replies
- 0 kudos
Databricks Genie One - Managing past conversation initiated by Databricks service principal account
Hello everyone.Recently, our team use Databricks service principal account, connecting to Genie One MCP server (via unity gateway) for data retrieval. However, we notice that there are currently no ways for us to manage the past conversations that ha...
- 0 Views
- 0 replies
- 0 kudos
- 885 Views
- 3 replies
- 1 kudos
How to Track Hourly or Daily # of Upsert/Delete Metrics in a DLT Streaming Pipeline
We created a Delta Live Tables (DLT) streaming pipeline to ingest data from the Bronze layer to the Silver layer using Change Data Feed (CDF) enabled.The stream runs continuously and shows # of upserted and deleted rows at an aggregate level from the...
- 885 Views
- 3 replies
- 1 kudos
Latest Reply
- 1 kudos
One distinction worth making before choosing the hourly bucket: _commit_timestamp measures when changes landed in the table, while a source event timestamp measures when the business event happened. Late arrivals make those different reports.For targ...
- 1 kudos
- 609 Views
- 4 replies
- 4 kudos
Resolved! At what data size do you stop reaching for Spark?
Hi everyone,A question I keep having with my team and I would like to hear how others think about it.A lot of the jobs we run are not big. Plenty of our pipelines process a few gigabytes, some considerably less. We run them on Spark because that is w...
- 609 Views
- 4 replies
- 4 kudos
Latest Reply
- 4 kudos
I would separate the engine decision from the compute-startup decision. One nuance on single-node Spark: it removes cross-machine shuffle traffic, but it doesn't remove every Exchange, local spill or startup cost.A useful comparison is the complete r...
- 4 kudos
- 3087 Views
- 9 replies
- 3 kudos
Resolved! schema evolution with structured streaming: upstream schema change causes downstream writer fails.
Hello,Bronze: use classic or job compute, Autoloader with.option("mergeSchema", "true"). Schema evolution works correctly. data goes to bronze.my_bronze_table.Silver: uses serverless compute, reader reads bronze.my_bronze_table, does all necessary t...
- 3087 Views
- 9 replies
- 3 kudos
Latest Reply
- 3 kudos
Your follow-up points to the join input as the boundary to stabilize, rather than just the final Silver schema.If the enrichment logic permits it, parse and unpivot before the stateful join, so new attributes become rows such as entity_key, attribute...
- 3 kudos
- 226 Views
- 3 replies
- 3 kudos
Resolved! Production table maintenance
Hi Community members,I'm interested in how people monitor the effectiveness of Liquid Clustering as a Delta table grows over time.Consider below scenario:A table initially performs well, but MERGE operations gradually become slower even though the cl...
- 226 Views
- 3 replies
- 3 kudos
Latest Reply
- 3 kudos
One correctness check on the pruning suggestion: adding a target-date filter to MERGE changes which rows can match.With ON t.id = s.id AND t.event_date >= batch_min_date, an existing key whose target date is older can become unmatched and get inserte...
- 3 kudos
- 115 Views
- 1 replies
- 2 kudos
Bypassing the Message Broker with Databricks Zerobus Ingest
Hi everyoneI recently explored the new Databricks Zerobus feature and built a hands-on demo project to test its efficiency. To capture the architectural shift.The Old Approach: Routing event streams through intermediate broker buses (like Kafka or Ki...
- 115 Views
- 1 replies
- 2 kudos
Latest Reply
- 2 kudos
Thanks for sharing the demo. I haven't benchmarked it myself, so I'll add what I'd want to check before dropping a broker, rather than numbers.Zerobus removes the broker when Databricks is the only consumer of those events. If another system also nee...
- 2 kudos
- 68 Views
- 1 replies
- 0 kudos
Jev as Judge for Genie Agents
Can we configure the Jev to be as judge for Genie Agents ? I know we can log traces of Genie Agents into ML Flow on top of it can we bring in Jev and score the outputs generated ?
- 68 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Yes, this is doable, and the pieces are already documented. Two parts.1. Get the Genie conversations into an MLflow experiment as traces, which you already know how to do. The MLflow cookbook page on Genie evaluation judges walks through tracing conv...
- 0 kudos
- 36 Views
- 0 replies
- 1 kudos
Why Your Genie Space Gives Wrong Answers: A Debugging Playbook
Every team that builds a Databricks Genie space goes through the same moment. The demo went well, the first few questions came back clean, and then a stakeholder asks something slightly different and the number is wrong. Not obviously wrong, which wo...
- 36 Views
- 0 replies
- 1 kudos
- 28 Views
- 0 replies
- 0 kudos
Feature Request: Cross-job dependencies with business-date awareness
SummaryWorkflows can chain tasks inside one job, but there is no native way to say: “Run job X once per day, after jobs A, B, C, D and E have each completed successfully for the same business date, even though they are independent jobs with their own...
- 28 Views
- 0 replies
- 0 kudos
- 1990 Views
- 7 replies
- 7 kudos
checkpoint changes not working on my databricks job
Hi,I do have a job processing kafka stream using kafka.readstream process and due to some issue we changed the checkpoint path to other path and it pulled all the records and later when i changed to the original checkpoint location it is not pulling ...
- 1990 Views
- 7 replies
- 7 kudos
Latest Reply
- 7 kudos
Stay on the new checkpoint. It already holds the latest offsets, so just keep running it. The only cleanup is the duplicates from the replay. Dedupe the raw table on topic, partition, offset if you store them, otherwise on your business key or a hash...
- 7 kudos
- 2783 Views
- 6 replies
- 7 kudos
Resolved! AutoLoader Type Widening
I have a few doubts regarding AutoLoader behavior and capabilities. Please check and correct wherever my assumptions or understanding are incorrect, much appreciated. Below is my specific code Example scenario:Target Managed Delta Table (Type Widenin...
- 2783 Views
- 6 replies
- 7 kudos
Latest Reply
- 7 kudos
Hi Parth, a useful update for anyone coming back to this thread: Auto Loader now supports type widening in Public Preview on DBR 16.4+ through cloudFiles.schemaEvolutionMode = addNewColumnsWithTypeWidening.Your current addNewColumns setting still doe...
- 7 kudos
- 221 Views
- 6 replies
- 10 kudos
Datavault implemention on Databricks
Hi everyone,I'm building a Data Vault 2.0 architecture entirely natively in Databricks from scratch. We have specifically decided not to use dbt.While Raw Vault ingestion is straightforward with Spark/Delta, I'm looking for architectural advice on co...
- 221 Views
- 6 replies
- 10 kudos
Latest Reply
- 10 kudos
Great points from @balajij8 on the SDP/LDP approach. I've worked on a few Data Vault implementations, and these are the things that tend to bite when the Business Vault grows:1. Hash keys: standardize before hashing. Trim, upper-case and use a fixed ...
- 10 kudos
- 223 Views
- 1 replies
- 1 kudos
Document AI is a pipeline, not a model
When a financial document needs to be approved or rejected, there usually isn't one model making the decision.In the check / payment-slip systems I worked on, the pipeline looked more like this:1. Multimodal models + OCR to extract fields2. YOLO to d...
- 223 Views
- 1 replies
- 1 kudos
Latest Reply
- 1 kudos
On your closing question, the rule I'd use for the line: anything with a checkable ground truth, either inside the document or in a system of record, goes to deterministic logic, and the model only proposes. Amount in figures vs words, date validity,...
- 1 kudos
- 40 Views
- 0 replies
- 0 kudos
Per-agent guardrails: what do you alert on, tokens, retries, or cost per task?
I'm trying to work out what a sensible guardrail looks like for each agent when several of them share a few serving endpoints.A plain token budget feels too blunt. An agent that retries a failing tool four times looks like a heavy user, while a cheap...
- 40 Views
- 0 replies
- 0 kudos
- 126 Views
- 2 replies
- 0 kudos
Enable non-DE teams to build their own pipelines
Hi! In all the latest dbks events I keep hearing this idea of shifting away from a centralized data team, and enabling the final consumer teams to build their own data pipelines with Lakeflow instead. This seems the worst idea ever to me, and sounds...
- 126 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
Your worry is fair, but I'd push on where the dividing line sits. I don't think it should be team role (DE vs analyst). I think it should be how many other people depend on the output.A table that only one analyst reads can be as fragile as they like...
- 0 kudos
Announcements
| User | Count |
|---|---|
| 2293 | |
| 1501 | |
| 1255 | |
| 1188 | |
| 961 |
Latest from our Blog
Locate, Measure, Judge: A Pattern for Verifiable Visual AI on Databricks (Part 2)
Part 1 introduced the pattern for verifiable visual AI. Let the model find the evidence, use deterministic code to measure it, apply governed rules to produce a pass, review, or fail verdict, and send...
Locate, Measure, Judge: A Pattern for Verifiable Visual AI on Databricks (Part 1)
AI adoption has moved much faster than measurable business results. McKinsey reports that 88% of organizations now use AI in at least one business function, yet only 39% see a bottom-line effect. A co...
Minimize disruptions for continuous pipelines with Maintenance Windows
Minimize disruptions for continuous pipelines with Maintenance Windows In an agentic world, it is more important than ever to make decisions using the latest available data. To achieve this, data engi...
Introducing session restore for serverless jobs
Serverless jobs make it easy to run production notebook workloads at scale, but until now, the state of a run was gone once it ended. If a pipeline ran for hours and failed on the last step, the varia...
