Hey Databricks Community! Join us for a free, community-sponsored BrickTalk on Thursday, September 17, 2026! BrickTalks is a community event series where Databricks experts share real-world use cases, demos, and practical insights for building with d...
- 218 Views
- 0 replies
- 2 kudos
Monday
REGISTRATION IS NOW OPEN The Data + AI World Tour is heading your way A free, one-day event on how organizations are building AI they can trust to deliver real outcomes - by grounding models in their own enterprise data and business context. THE...
- 567 Views
- 0 replies
- 2 kudos
a week ago
Databricks Advanced Learning Festival September 16 – October 14, 2026 Join us for a four-week event dedicated to learning, upskilling, and advancing your career in data engineering, analytics, machine learning, and generative AI. Whether you are...
- 41122 Views
- 46 replies
- 34 kudos
3 weeks ago
Virtual Event Time: AMER Sept 23 / 9 AM PTEMEA Sept 24 / 9 AM BST / 10 AM CESTAPJ Sept 24 / 9:30 AM IST / 12 PM SGT / 1 PM JST/KST / 2 PM AEST (Japanese and Korean captions available) Register for the Workshop Note: Marking RSVP on this Community ...
- 401 Views
- 0 replies
- 2 kudos
a week ago
Databricks EMEA Learning Festival Hands On Virtual Training 29–30 September 2026 · 09:00 BST / 10:00 CEST · Fully virtual · Free to join Ready to accelerate your journey on Data & AI? Join us for the EMEA Learning Festival. A two-day, hands-on vir...
- 2523 Views
- 0 replies
- 2 kudos
a month ago
Subscriber attrition can be difficult to anticipate. The Subscriber Churn Prediction Solution Accelerator shows how to analyze behavioral data, identify subscribers at increased risk of cancellation, and use machine learning to estimate churn likelih...
- 373 Views
- 1 replies
- 5 kudos
a week ago
The Weekly Digest • August 31 – September 6 Community Pulse Your weekly pulse: new articles, real answers, and the discussions worth watching. This Week's Community Stars This week's difference-makers – and a shout-out to the new members ear...
- 181 Views
- 0 replies
- 1 kudos
Tuesday
Community Activity
- 2063 Views
- 3 replies
- 1 kudos
Resolved! Request Access for Notebooks or workspaces from Databricks Community Edition
I have been a long-time user of Databricks Community Edition, where I created and maintained several notebooks for learning and experimentation.Recently, I understand that the legacy Community Edition has been migrated to the new Free Edition. Since ...
- 2063 Views
- 3 replies
- 1 kudos
- 24 Views
- 0 replies
- 0 kudos
Materialized Views vs Streaming Tables in Databricks: A Practical Guide for Data Engineers
As data platforms move toward more declarative and incremental processing, one question comes up frequently when building pipelines in Databricks:Should I use a Materialized View or a Streaming Table?Both are important building blocks in Lakeflow pip...
- 24 Views
- 0 replies
- 0 kudos
- 20 Views
- 0 replies
- 0 kudos
databricks-langchain PyPI installation stuck for 30+ minutes on multiple Azure Databricks clusters
Team,I am experiencing an issue installing the databricks-langchain Python package from PyPI on Azure Databricks.The package installation remains in the installing/resolving state for more than 30 minutes and does not complete.Testing performed:Origi...
- 20 Views
- 0 replies
- 0 kudos
- 49 Views
- 0 replies
- 0 kudos
Announcement | How we eliminated $1 million a year of wasted AI agent spend in one hour
Silent MCP tool failures can make agents look productive while quietly increasing token usage and wait time. By combining Unity Gateway tracing with Genie One, Databricks engineers turned that hidden waste into a ranked, fixable bug list, identifying...
- 49 Views
- 0 replies
- 0 kudos
- 23 Views
- 0 replies
- 0 kudos
shutil.copy from /local_disk0 to Unity Catalog Volume hangs for hours — recommended pattern for log
SetupSingleton Python logger (logging.FileHandler) writing .log files during ADF-orchestrated notebook runs (one notebook per file). Files need to land in a UC Volume.Context on the migration path — this workflow has been reshaped repeatedly by the m...
- 23 Views
- 0 replies
- 0 kudos
- 29 Views
- 1 replies
- 0 kudos
How are you combining forecasting models for time-series work?
Most time-series forecasting jobs do not come down to one model. Teams run N-BEATS, NHITS, LSTM, and GRU, then struggle to combine them well.I want to open a discussion on model synthesis. Not data cleaning. Not generative AI. I mean merging several ...
- 29 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi,Good topic. A few things that have held up for us.On weighting across horizons, fit weights per horizon step rather than one global set, and derive them from rolling origin backtests so the errors are genuinely out of sample. Models tend to trade ...
- 0 kudos
- 56 Views
- 2 replies
- 2 kudos
Resolved! Auto Loader Strategy: Balancing Cloud Notification Costs vs. Directory Listing in Massive Migrations
Hi everyone,I’m currently architecting a high-volume migration for a platform that generates millions of small files daily across thousands of prefixes. As I evaluate the best ingestion strategy using Databricks Auto Loader, I’m running into an archi...
- 56 Views
- 2 replies
- 2 kudos
Latest Reply
- 2 kudos
In the context of enterprise-grade cloud migrations (think legacy databases or petabyte-scale data lakes ingestion into Databricks Delta Lake), the industry standard approach involves splitting the ingestion stack into 2:History (Initial Bulk Load) a...
- 2 kudos
- 49 Views
- 2 replies
- 0 kudos
Resolution of Missing Certificates in Accredible
I have successfully completed all the required certifications and courses for the Databricks Champion Program. All the certificates and badges were issues but these certifications are not visible in my Accredible account associated with username@comp...
- 49 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi,Sorry, that is a frustrating place to be stuck.The thing that will unblock you fastest is separating the two problems. The account merge is one issue, but the panel discussion does not actually depend on it. Every Accredible credential has its own...
- 0 kudos
- 42 Views
- 2 replies
- 2 kudos
Resolved! Managing Schema Drift and Evolution in Spark Declarative Pipelines (SDP)
Hi everyone,I’m currently diving into the Spark Declarative Pipelines (SDP) framework and evaluating it for our upcoming migration of a high-velocity event-driven pipeline.One area I’m trying to solidify is our strategy for Schema Evolution. In tradi...
- 42 Views
- 2 replies
- 2 kudos
Latest Reply
- 2 kudos
Hi,One framing correction that might save you time: there is no separate schema drift control at the declarative layer. SDP handles orchestration and dependencies between flows. Schema evolution is still owned by the source reader and by the target t...
- 2 kudos
- 87 Views
- 2 replies
- 1 kudos
DELETE Removed the Customer Record—but Did It Remove the Data?
DELETE Removed the Customer Record—but Did It Remove the Data?A successful Delta Lake DELETE makes a record disappear from the current table version. However, it may not immediately remove that record from the underlying storage files.Imagine a ficti...
- 87 Views
- 2 replies
- 1 kudos
Latest Reply
- 1 kudos
Good breakdown, and the logical versus physical distinction is the part most teams get wrong. Three gaps worth adding to that runbook.Change data feed. If it is enabled on the table, the deleted rows also exist in the _change_data directory as change...
- 1 kudos
- 150 Views
- 5 replies
- 2 kudos
Resolved! Do UC table tags propagate to billing for Predictive Optimization and Data Quality Monitoring?
I am looking at a scenario for team-level cost attribution using system.billing.usage and have a tagging gap for Databricks-managed services such as Predictive Optimization and Data Quality Monitoring.Tried adding a team/ownership tag directly to the...
- 150 Views
- 5 replies
- 2 kudos
Latest Reply
- 2 kudos
@stbjelcevic Thanks, that was helpful and gave another way to solve this use case.Instead of maintaining a separate table for object to team mapping table, can now use UC tags as the ownership source with precedence:table tag -> schema tag -> catalog...
- 2 kudos
- 144 Views
- 6 replies
- 4 kudos
Resolved! How are you separating dev, staging and prod in Unity Catalog without duplicating everything?
Hi everyone, We are trying to settle on an environment strategy and keep going back and forth, so I would like to hear how other teams actually landed on this rather than what the reference architecture suggests. Our situation. One Databricks account...
- 144 Views
- 6 replies
- 4 kudos
Latest Reply
- 4 kudos
Here is, in summary, how I am facing that issue in some real projects in Deloitte:Unity Catalog Metastore│├── Workspace DEV│ └── sales_dev│ └── finance_dev│ └── customer_dev│├── Workspace QA│ └── sales_qa│ └── finance_qa│ └── customer_qa│├── Workspac...
- 4 kudos
- 25 Views
- 0 replies
- 0 kudos
Databricks Support -0100505- Exam Suspended because non compliance requirements set by test sponsor
@Cert-Team @Cert-TeamOPS I am writing to urgently escalate my existing support case #01005055 regarding my Databricks certification exam being suspended due to an alleged non-compliance issue.I raised the case promptly and have already followed up, b...
- 25 Views
- 0 replies
- 0 kudos
- 57 Views
- 1 replies
- 1 kudos
BOOTSTRAP_TIMEOUT on cluster start (Southeast Asia)
Our cluster start fails with:BOOTSTRAP_TIMEOUT: Instance bootstrap failed command: Bootstrap_e2e Instance bootstrap inferred timeout reason: UnknownReason Failure message: Bootstrap is terminated spontaneously by VM because self timeout was reached. ...
- 57 Views
- 1 replies
- 1 kudos
Latest Reply
- 1 kudos
Widening the NSG service tag to all regions is a good first step but it rarely fixes this on its own. The AzureDatabricks service tag only covers the control plane traffic. What most people miss is that during bootstrap, the VM also needs to reach Az...
- 1 kudos
- 66 Views
- 1 replies
- 3 kudos
Resolved! Now that Unity Catalog manages Iceberg natively, is anyone actually switching?
Hi everyone,Unity Catalog managed Iceberg tables went GA earlier this year, and they arrived with the features that used to be the reason to stay on Delta: liquid clustering, predictive optimization, materialized views, streaming tables. On paper the...
- 66 Views
- 1 replies
- 3 kudos
Latest Reply
- 3 kudos
Hi @Islam_hoti,The Reality on the Ground: We have experimented with Iceberg, but for our primary production workloads, we remain 100% Delta Lake. Here is our internal "Messy Truth" on why:The "Inertia" Factor: You’re right, inertia is a legitimate en...
- 3 kudos
Announcements
| User | Count |
|---|---|
| 2293 | |
| 1401 | |
| 1253 | |
| 1186 | |
| 961 |
Latest from our Blog
Introducing Stream-Stream Join Support in Apache Spark Real-Time Mode
Summary Explore how Real-Time Mode for Apache Spark Structured Streaming on Databricks now supports inner stream-stream joins in update mode — with sub-second latency.See it in action on a real-time a...
Building Stateful Agents on Lakebase
TL;DR — Production agents need more than conversation history. They must survive restarts, recall decisions from earlier sessions, and work with current business data. In our Supply Chain Planner Copi...
Lakeflow Connect: Message Bus Ingestion - Now shipping your logs directly! (Beta)
Take a quick inventory of where your operational data lives right now. There is probably a Kafka cluster carrying your event backbone, or perhaps RabbitMQ capturing messages on a factory floor or bet...
GeoGenie 2: Ask your map a question in plain English
A year ago we put a natural-language layer on a 3D map. This time we rebuilt the whole thing on Lakebase and PostGIS, wired Genie to the same tables, and made the map answer back, drawn region and all...
