The Genie has spoken And the winners of the Genie-Powered App Challenge are… A few weeks ago, we threw down a challenge: build something incredible with Databricks Apps and Genie Agent. And wow, you delivered. From smart, real-world prob...
- 613 Views
- 8 replies
- 6 kudos
yesterday
Hey Databricks Community! Join us for a free, community-sponsored BrickTalk on Thursday, September 17, 2026! BrickTalks is a community event series where Databricks experts share real-world use cases, demos, and practical insights for building with d...
- 285 Views
- 0 replies
- 2 kudos
Monday
REGISTRATION IS NOW OPEN The Data + AI World Tour is heading your way A free, one-day event on how organizations are building AI they can trust to deliver real outcomes - by grounding models in their own enterprise data and business context. THE...
- 621 Views
- 0 replies
- 2 kudos
a week ago
Databricks Advanced Learning Festival September 16 – October 14, 2026 Join us for a four-week event dedicated to learning, upskilling, and advancing your career in data engineering, analytics, machine learning, and generative AI. Whether you are...
- 46204 Views
- 59 replies
- 38 kudos
3 weeks ago
Virtual Event Time: AMER Sept 23 / 9 AM PTEMEA Sept 24 / 9 AM BST / 10 AM CESTAPJ Sept 24 / 9:30 AM IST / 12 PM SGT / 1 PM JST/KST / 2 PM AEST (Japanese and Korean captions available) Register for the Workshop Note: Marking RSVP on this Community ...
- 463 Views
- 0 replies
- 2 kudos
a week ago
Databricks EMEA Learning Festival Hands On Virtual Training 29–30 September 2026 · 09:00 BST / 10:00 CEST · Fully virtual · Free to join Ready to accelerate your journey on Data & AI? Join us for the EMEA Learning Festival. A two-day, hands-on vir...
- 2606 Views
- 0 replies
- 2 kudos
08-11-2026
Product onboarding can be complex and time-consuming when teams work with inconsistent data formats, inaccurate information and multiple stakeholders. The Simplifying Product Onboarding With Generative AI Solution Accelerator shows how generative AI ...
- 76 Views
- 0 replies
- 0 kudos
Tuesday
The Weekly Digest • August 31 – September 6 Community Pulse Your weekly pulse: new articles, real answers, and the discussions worth watching. This Week's Community Stars This week's difference-makers – and a shout-out to the new members ear...
- 236 Views
- 0 replies
- 1 kudos
Tuesday
Community Activity
- 64 Views
- 1 replies
- 0 kudos
OAuth M2M (client-credentials) - getting error with github run
I'm setting up OAuth M2M (client-credentials) authentication for a service principal, as the long-term replacement for PAT-based auth in a GitHub Actions CI/CD workflow. The token exchange itself succeeds and returns a valid, well-formed access token...
- 64 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
further refined this problem with below if anyone has any idea how I could address? thanks. Later in troubleshooting, this same should_change_password error started appearing on a personal-account PAT as well (not just the SP's OAuth token) — so what...
- 0 kudos
- 42 Views
- 1 replies
- 1 kudos
Best Practice for Handling Schema Evolution with Auto Loader in Production?
Hi Databricks Community,I’d like to hear how other Data Engineers are handling schema evolution with Auto Loader in production environments.Consider the following scenario:We have a continuously running ingestion pipeline using Auto Loader that proce...
- 42 Views
- 1 replies
- 1 kudos
Latest Reply
- 1 kudos
Hi @Islam_hoti,This is a central design decision for any production-grade Lakehouse. In my experience architecting petabyte-scale environments, the goal is to minimize "Pipeline Friction" without sacrificing "Data Governance."Here is the "Architectur...
- 1 kudos
- 22 Views
- 0 replies
- 0 kudos
Announcement | Autoscaling Lakebase Postgres
Choosing a fixed database size before you know the workload is an old pattern. Lakebase Postgres removes that sizing exercise by continuously adjusting compute to match demand, while keeping PostgreSQL running and the data safely decoupled in the sto...
- 22 Views
- 0 replies
- 0 kudos
- 33 Views
- 1 replies
- 0 kudos
Getting started with Delta Lake basics
Hi all — I am working through the Delta Lake intro and wanted to share a short note on using OPTIMIZE after merge. Happy to hear better patterns.
- 33 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Did you forgot to share the short note? If you are referring to running optimize after Merge, its worth mentioning that it doesn't need to be run after every Merge.MERGE can create or rewrite files, but whether OPTIMIZE is needed immediately depends ...
- 0 kudos
- 58 Views
- 2 replies
- 0 kudos
Auto Loader stream fails on RocksDB checkpoint after enabling managed file events
Hello Community.We recently enabled the following option on several existing Auto Loader streams:"cloudFiles.useManagedFileEvents": "true"Most streams continued working, but one now fails while restoring its existing checkpoint, referencing a file si...
- 58 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi @data_pulse I had a quick look at this problem alongside the Databricks documentation and found a supported way to recover your stream without resetting your entire checkpoint!This error usually happens when Auto Loader tries to initialize its sta...
- 0 kudos
- 23 Views
- 0 replies
- 0 kudos
Changing Root Metastore Location
We currently have a default root metastore location linked to an ADLS container.At the same time we are working on a ADLS migration project.All of the tables and files (everything we use for day to day business related) have moved to the new ADLS so ...
- 23 Views
- 0 replies
- 0 kudos
- 28 Views
- 0 replies
- 0 kudos
CSV export from AI/BI Dashboard table widget does not preserve visual column order
SummaryWhen downloading data as CSV from a table widget on an AI/BI Dashboard (kebab menu → Download → Data → Download CSV), the column order in the exported file does not match the column order displayed in the dashboard. All aggregated (measure) co...
- 28 Views
- 0 replies
- 0 kudos
- 47 Views
- 1 replies
- 0 kudos
Controlling allow/ask/deny lists for predelivered MCP connectors
Hello,We are looking at utilizing the pre-delivered MCP connector to M365 via Genie One. The connector works great, but each user can individually configure the allowable tool calls without any central governance layer.Are we missing something or is ...
- 47 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Hey @Leembo system.ai.microsoft_365 is now a 1st class UC object ("MCP service") that runs through Unity Gateway, same as the external connectors you register yourself. Two levels of control: - Whole connector on/off: an EXECUTE grant, so you can al...
- 0 kudos
- 56 Views
- 1 replies
- 0 kudos
How to create monotonic function to incrementally add obj_id for datasource in pyspark
I have used monotonically_increasing_id() to add unique to datasource .However , this does not add unique id in increasing number ..Its kind of random.How to perform this in pyspark.
- 56 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Hey So monotonically_increasing_id() only promises the numbers go up and are unique, not that they're neat like 1, 2, 3. Under the hood it bakes the partition number into the ID, so each partition jumps ahead by billions.If you let me know what you'...
- 0 kudos
- 28 Views
- 0 replies
- 0 kudos
Storage Credential creation fails with "Access Connector ... could not be found"
Subject: Storage Credential creation fails with "Access Connector ... could not be found" despite fully verified Azure configurationHi all,I'm trying to create a storage credential (Azure Managed Identity) for Unity Catalog on a Premium Azure Databri...
- 28 Views
- 0 replies
- 0 kudos
- 33 Views
- 0 replies
- 0 kudos
Free Trial – “Daily limit for workspace creation” error
Hi everyone,I’m currently using a Databricks Free Trial on Google Cloud and I'm trying to create a workspace.The workspace creation fails with the following error:RESOURCE_EXHAUSTED: Limit reached for workspace creation. You've hit your daily limit f...
- 33 Views
- 0 replies
- 0 kudos
- 188 Views
- 2 replies
- 1 kudos
How Can I Become a Databricks Partner Champion Through Technical Expertise?
Hello Databricks Community,I am actively building my technical expertise in the Databricks ecosystem and would like guidance on the roadmap to becoming a Databricks Partner Champion, with a strong focus on technical skills and hands-on implementation...
- 188 Views
- 2 replies
- 1 kudos
Latest Reply
- 1 kudos
Edit: More information in one place What are the current eligibility requirements for the Databricks Partner Champion Program?You need to meet the prerequisites, including 2 Fundamental certifications (Data + Generative AI Fundamentals), 1 Associate-...
- 1 kudos
- 28 Views
- 0 replies
- 0 kudos
How do you test whether a retriever stops too early?
Hello everyone,Databricks’ Adaptive Instructed-Retriever announcement describes a model that learns when to stop searching. How would you test whether it stops after finding a general policy but before finding the exception that changes the answer?I...
- 28 Views
- 0 replies
- 0 kudos
- 27 Views
- 0 replies
- 0 kudos
Building for Failure: Implementing Data Quality Firewalls in Petabyte-Scale Medallion Architectures
In my 13 years of architecting data platforms—from Retail to Industrial Gas Power—I’ve learned one universal truth: A fast pipeline that delivers bad data is just a liability.As we move toward Declarative Pipelines (DLT), the role of the Architect sh...
- 27 Views
- 0 replies
- 0 kudos
- 77 Views
- 1 replies
- 2 kudos
Resolved! Designing an Effective "Quarantine" Pattern for Failed DLT Expectations
Hi everyone,I’m building out a production Medallion Architecture using Delta Live Tables (DLT), and I’m refining our strategy for handling data that fails Expectations.While expect_or_drop and expect_or_fail are straightforward, I’m struggling to fi...
- 77 Views
- 1 replies
- 2 kudos
Latest Reply
- 2 kudos
Hi,Good questions. A few things that have worked.On the quarantine pattern, the version most people land on first is two flows from the same source, one with the expectations and one with the inverted condition. It works, but it reads the source twic...
- 2 kudos
Announcements
| User | Count |
|---|---|
| 2293 | |
| 1406 | |
| 1253 | |
| 1186 | |
| 961 |
Latest from our Blog
Actually understanding Unity Catalog Managed Tables
The “why” worth knowing Choosing external tables is usually about flexibility: you keep dedicated storage, maintain raw files that multiple engines can query directly, and control your own partitionin...
Introducing Stream-Stream Join Support in Apache Spark Real-Time Mode
Summary Explore how Real-Time Mode for Apache Spark Structured Streaming on Databricks now supports inner stream-stream joins in update mode — with sub-second latency.See it in action on a real-time a...
Building Stateful Agents on Lakebase
TL;DR — Production agents need more than conversation history. They must survive restarts, recall decisions from earlier sessions, and work with current business data. In our Supply Chain Planner Copi...
Lakeflow Connect: Message Bus Ingestion - Now shipping your logs directly! (Beta)
Take a quick inventory of where your operational data lives right now. There is probably a Kafka cluster carrying your event backbone, or perhaps RabbitMQ capturing messages on a factory floor or bet...
