The Genie has spoken And the winners of the Genie-Powered App Challenge are… A few weeks ago, we threw down a challenge: build something incredible with Databricks Apps and Genie Agent. And wow, you delivered. From smart, real-world prob...
- 1076 Views
- 9 replies
- 7 kudos
Thursday
Hey Databricks Community! Join us for a free, community-sponsored BrickTalk on Thursday, September 17, 2026! BrickTalks is a community event series where Databricks experts share real-world use cases, demos, and practical insights for building with d...
- 442 Views
- 0 replies
- 2 kudos
a week ago
REGISTRATION IS NOW OPEN The Data + AI World Tour is heading your way A free, one-day event on how organizations are building AI they can trust to deliver real outcomes - by grounding models in their own enterprise data and business context. THE...
- 775 Views
- 0 replies
- 2 kudos
2 weeks ago
Databricks Advanced Learning Festival September 16 – October 14, 2026 Join us for a four-week event dedicated to learning, upskilling, and advancing your career in data engineering, analytics, machine learning, and generative AI. Whether you are...
- 59504 Views
- 83 replies
- 40 kudos
4 weeks ago
Virtual Event Time: AMER Sept 23 / 9 AM PTEMEA Sept 24 / 9 AM BST / 10 AM CESTAPJ Sept 24 / 9:30 AM IST / 12 PM SGT / 1 PM JST/KST / 2 PM AEST (Japanese and Korean captions available) Register for the Workshop Note: Marking RSVP on this Community ...
- 663 Views
- 0 replies
- 2 kudos
2 weeks ago
Databricks EMEA Learning Festival Hands On Virtual Training 29–30 September 2026 · 09:00 BST / 10:00 CEST · Fully virtual · Free to join Ready to accelerate your journey on Data & AI? Join us for the EMEA Learning Festival. A two-day, hands-on vir...
- 2859 Views
- 0 replies
- 3 kudos
08-11-2026
Community Champion • Our Community Champion Program celebrates members who consistently contribute their expertise, support fellow practitioners, and help shape a stronger and more collaborative Databricks Community. Every month, we recognize individ...
- 201 Views
- 5 replies
- 5 kudos
yesterday
Product onboarding can be complex and time-consuming when teams work with inconsistent data formats, inaccurate information and multiple stakeholders. The Simplifying Product Onboarding With Generative AI Solution Accelerator shows how generative AI ...
- 184 Views
- 0 replies
- 0 kudos
a week ago
The Weekly Digest • August 31 – September 6 Community Pulse Your weekly pulse: new articles, real answers, and the discussions worth watching. This Week's Community Stars This week's difference-makers – and a shout-out to the new members ear...
- 406 Views
- 1 replies
- 2 kudos
a week ago
Community Activity
- 27 Views
- 2 replies
- 0 kudos
Lakeflow connect CT pipeline keeps running multiple connections in source
I'm working on an ingestion pipeline. I was adding tables to it gradually so that we can monitor the load on our prod server side by side and with my last set of tables added (with them, all the heavy duty tables were inside that pipeline), the load ...
- 27 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi @Ashwin_DSA, Thankyou for responding. In your suggested solutions, you mentioned "Downsize the gateway driver node. The gateway runs entirely on the driver node of a classic compute cluster. The parallelism is tied to the number of CPU cores on th...
- 0 kudos
- 201 Views
- 5 replies
- 15 kudos
Databricks Community Champion - August 2026 - Shamen Paris
Community Champion • Our Community Champion Program celebrates members who consistently contribute their expertise, support fellow practitioners, and help shape a stronger and more collaborative Databricks Community. Every month, we recognize individ...
- 201 Views
- 5 replies
- 15 kudos
- 59 Views
- 2 replies
- 2 kudos
Resolved! ConcurrentAppendException on parallel MERGE into the same Delta table, despite disjoint keys
Hi everyone, Hoping someone has been through this, because I am running out of ideas. Setup. DBR 15.4 LTS, Unity Catalog managed Delta table, roughly 900 million rows. Six upstream sources each write into the same target table through their own job, ...
- 59 Views
- 2 replies
- 2 kudos
Latest Reply
- 2 kudos
Hi @Islam_hoti This is a classic "Optimistic Concurrency Control" bottleneck in Delta Lake. Even with partitions, the MERGE operation requires a metadata-level lock on the table's transaction log, which is where your contention is happening.Here is a...
- 2 kudos
- 29 Views
- 0 replies
- 0 kudos
A Metadata-Driven Backdated Extraction Pattern for IoT/Sensor Pipelines Using Zerobus and Databricks
IntroductionIoT devices generate a large amount of data every day. In a perfect scenario, all this data arrives on time and our ingestion pipeline processes it using incremental loads. But in real-world IoT systems, this does not always happen.A devi...
- 29 Views
- 0 replies
- 0 kudos
- 28 Views
- 0 replies
- 0 kudos
Solution Accelerator Series | Customer Entity Resolution
Building a customer 360 requires connecting customer records across disparate data sets and establishing common customer identities. The Customer Entity Resolution Solution Accelerator shows how to build that foundation by translating customer attrib...
- 28 Views
- 0 replies
- 0 kudos
- 408 Views
- 4 replies
- 4 kudos
Resolved! Difference between Workspace and Unity Catalog experiments when using MLflow autologging?
Hi everyone,I am trying to understand the exact differences between using Workspace experiments versus Unity Catalog experiments, specifically in the context of MLflow autologging (mlflow.autolog()).Does autologging behave differently depending on wh...
- 408 Views
- 4 replies
- 4 kudos
Latest Reply
- 4 kudos
Great question! I’d also be interested to know if autologging has any behavioral or permission differences with Unity Catalog experiments, especially around governance and model lineage.
- 4 kudos
- 951 Views
- 2 replies
- 1 kudos
Data practitioner in AI Era
As the AI revolution takes off in 2025, there is a renewed emphasis on adopting a Data-First approach. Organizations are increasingly recognizing the need to establish a robust data foundation while preparing a skilled fleet of Data Engineers to tack...
- 951 Views
- 2 replies
- 1 kudos
Latest Reply
- 1 kudos
Absolutely agree! AI is only as powerful as the data behind it. The evolving role of data engineers in building that foundation is definitely worth discussing.
- 1 kudos
- 59 Views
- 2 replies
- 4 kudos
Resolved! AUTO CDC FROM SNAPSHOT schema evolution in history tracking columns
Does the create_auto_cdc_from_snapshot_flow() allow for expansion of included columns in track_history_column_list or track_history_except_column_list without doing a full refresh? My intended flow is: Bronze - Managed ingestion of table (scd 1) usin...
- 59 Views
- 2 replies
- 4 kudos
Latest Reply
- 4 kudos
Adding a column and changing the history-tracking configuration need separate checks. Databricks describes adding columns as generally safe without a full refresh:"Plan your schema to avoid changes that require a full refresh. Adding columns is gener...
- 4 kudos
- 45 Views
- 2 replies
- 2 kudos
Slow Running SQL Query
The following Query is running slowly even though it returns only ~ a million rows. It takes around 25 mins. I am trying Produce a pre-aggregated summary table (grouped by 9 dimension attributes) from a large fact table, to be consumed by a Power BI ...
- 45 Views
- 2 replies
- 2 kudos
Latest Reply
- 2 kudos
With 1.4B rows, I’d first check join cardinality and whether filtering happens before the joins. Early reduction of the fact table could make a huge difference, especially before the final GROUP BY.
- 2 kudos
- 115 Views
- 4 replies
- 2 kudos
Compatibility Mode for AWS Redshift
Hello,I have several tables loaded from S3 using Lakeflow pipelines which generates streaming table and materialized views. Want to be able to read these tables in AWS redshift.Based on the Databricks documentation for compatibility mode:Databricks C...
- 115 Views
- 4 replies
- 2 kudos
Latest Reply
- 2 kudos
@HZ your understanding of Spectrum's Delta reader is correct: it requires manifests. The alternative is Redshift's Iceberg access through Glue catalog federation, which AWS documents for Unity Catalog. These are separate integration paths.For your La...
- 2 kudos
- 86 Views
- 2 replies
- 2 kudos
Feature enablement for Foundation Model Unity Catalog permissions
I am trying to restrict workspace users' access to Databricks Foundation Models using the guidance in the Foundation Model Unity Catalog Permissions documentation.I have revoked EXECUTE permission for all users from the system.ai schema. However, wor...
- 86 Views
- 2 replies
- 2 kudos
Latest Reply
- 2 kudos
The reason the two calls behave differently is that databricks-* and system.ai.* are separate serving and authorisation paths, and only the system.ai.* path runs through Unity Catalog. - system.ai.claude-sonnet-4-5 resolves to a UC model service, so ...
- 2 kudos
- 491 Views
- 2 replies
- 0 kudos
Resolved! post being marked as spam
Hi, My databricks app challenege is being tagged as SPAM. Please help.https://community.databricks.com/t5/community-articles/genie-powered-app-challege-bushfire-exposure-across-victoria-s/m-p/166804#M1481
- 491 Views
- 2 replies
- 0 kudos
- 39 Views
- 1 replies
- 0 kudos
Name mismatch confusion
Hi Databricks Team,I am registering for the Databricks Data Engineer Associate exam via Webassessor. My government ID reads '[First Name] [Father's Name] [Surname]', but Webassessor only offers a First and Last name field. Can you please confirm the ...
- 39 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
@teeshakshatriya , Try this Due to the fact that Kryterion/Webassessor profiles do not have a specific field for the Middle Name, make sure that you enter your father's name in the First Name field along with your own name.Correct Profile MappingFirs...
- 0 kudos
- 100 Views
- 3 replies
- 2 kudos
Failed to resolve External API
I am currently testing on a Premium trial, but my workspace is unable to resolve my external API to pull data. May I know if I need to update anything on the settings side to fix this issue?Thanks!
- 100 Views
- 3 replies
- 2 kudos
Latest Reply
- 2 kudos
This is usually related to network access rather than the Premium trial itself. I’d check the workspace’s serverless/external network access settings and make sure the API domain is allowed, then verify DNS resolution and any firewall or IP allowlist...
- 2 kudos
- 65 Views
- 0 replies
- 0 kudos
Databricks voucher
When can I see associate data bricks certification vouchers So that I can prepare and attend exam
- 65 Views
- 0 replies
- 0 kudos
Announcements
| User | Count |
|---|---|
| 2293 | |
| 1409 | |
| 1253 | |
| 1186 | |
| 961 |
Latest from our Blog
Tutorial: Transform your Lakeflow Connect ad data into visual and conversational analytics
Introduction Traditionally, unifying requires building brittle custom integrations for every single source. Lakeflow Connect resolves this with native, fully managed connectors for a full suite of mar...
Meet SDP Rewind: An undo button for your ETL pipelines
A bad deployment slips into your pipeline on Friday, and by Monday, it has been writing incorrect data for three days, with no clean way to roll it back. You can restore one table, but a pipeline is a...
Actually understanding Unity Catalog Managed Tables
The “why” worth knowing Choosing external tables is usually about flexibility: you keep dedicated storage, maintain raw files that multiple engines can query directly, and control your own partitionin...
Introducing Stream-Stream Join Support in Apache Spark Real-Time Mode
Summary Explore how Real-Time Mode for Apache Spark Structured Streaming on Databricks now supports inner stream-stream joins in update mode — with sub-second latency.See it in action on a real-time a...
