The Genie has spoken And the winners of the Genie-Powered App Challenge are… A few weeks ago, we threw down a challenge: build something incredible with Databricks Apps and Genie Agent. And wow, you delivered. From smart, real-world prob...
- 1479 Views
- 11 replies
- 9 kudos
a week ago
Hey Databricks Community! Join us for a free community-sponsored BrickTalk on Thursday, September 24, 2026. BrickTalks is a community event series where Databricks experts share real-world use cases, demos, and practical insights for building with Da...
- 197 Views
- 1 replies
- 2 kudos
Monday
REGISTRATION IS NOW OPEN The Data + AI World Tour is heading your way A free, one-day event on how organizations are building AI they can trust to deliver real outcomes - by grounding models in their own enterprise data and business context. THE...
- 968 Views
- 0 replies
- 3 kudos
3 weeks ago
Databricks Advanced Learning Festival September 16 – October 14, 2026 Join us for a four-week event dedicated to learning, upskilling, and advancing your career in data engineering, analytics, machine learning, and generative AI. Whether you are...
- 112490 Views
- 163 replies
- 54 kudos
a month ago
Virtual Event Time: AMER Sept 23 / 9 AM PTEMEA Sept 24 / 9 AM BST / 10 AM CESTAPJ Sept 24 / 9:30 AM IST / 12 PM SGT / 1 PM JST/KST / 2 PM AEST (Japanese and Korean captions available) Register for the Workshop Note: Marking RSVP on this Community ...
- 819 Views
- 0 replies
- 2 kudos
2 weeks ago
Databricks EMEA Learning Festival Hands On Virtual Training 29–30 September 2026 · 09:00 BST / 10:00 CEST · Fully virtual · Free to join Ready to accelerate your journey on Data & AI? Join us for the EMEA Learning Festival. A two-day, hands-on vir...
- 3178 Views
- 0 replies
- 3 kudos
08-11-2026
Community Champion • Our Community Champion Program celebrates members who consistently contribute their expertise, support fellow practitioners, and help shape a stronger and more collaborative Databricks Community. Every month, we recognize individ...
- 551 Views
- 6 replies
- 6 kudos
Monday
Building a customer 360 requires connecting customer records across disparate data sets and establishing common customer identities. The Customer Entity Resolution Solution Accelerator shows how to build that foundation by translating customer attrib...
- 162 Views
- 0 replies
- 2 kudos
Tuesday
The Weekly Digest • September 7 – September 13 Community Pulse A new week, a fresh wave of articles, answers, and discussions. Here's your weekly catch-up. This Week's Community Stars The members who showed up and made a difference this week ...
- 153 Views
- 0 replies
- 4 kudos
Thursday
Community Activity
- 77 Views
- 2 replies
- 0 kudos
Predictive Optimization for Streaming tables in Lakeflow pipelines
On July 2025 https://www.databricks.com/blog/whats-new-lakeflow-declarative-pipelines-july-2025 predictive optimization was enabled for all UC managed Lakeflow pipelines. I was wondering if there is a possibility now to disable it for Streaming table...
- 77 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi @Gilk, Before the how, can I ask what is driving the wish to turn it off? For most pipeline tables predictive optimization is doing useful work, running OPTIMIZE, VACUUM, and ANALYZE on serverless compute so file sizes, storage, and statistics sta...
- 0 kudos
- 181 Views
- 4 replies
- 0 kudos
Lakeflow connect CT pipeline keeps running multiple connections in source
I'm working on an ingestion pipeline. I was adding tables to it gradually so that we can monitor the load on our prod server side by side and with my last set of tables added (with them, all the heavy duty tables were inside that pipeline), the load ...
- 181 Views
- 4 replies
- 0 kudos
Latest Reply
- 0 kudos
For a standard connector, the gateway continuously extracts snapshots, change logs, and metadata, while a separately scheduled serverless pipeline applies staged data (connector components). Changing the serverless pipeline schedule therefore does no...
- 0 kudos
- 25 Views
- 0 replies
- 0 kudos
Power of Genie
Folks, I must admit, at the beginning I was apprehensive of using Genie for almost anything. Now it solves almost anything that I want to code on Databricks. I was more fascinated with it when I learnt that our customer (who is one of the largest Tel...
- 25 Views
- 0 replies
- 0 kudos
- 63 Views
- 2 replies
- 0 kudos
Databricks Free Edition
Hi,I just signed up for Databricks free edition to start my learning journey in Data Engineering on it.Want to know for how long it remains free?Thank you.
- 63 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi @kabiromohd, Welcome to Databricks, and a great choice for starting your Data Engineering journey! The short answer is that Databricks Free Edition is forever free. There is no trial period or expiration date. You can use it indefinitely as lo...
- 0 kudos
- 207 Views
- 2 replies
- 1 kudos
Changing Root Metastore Location
We currently have a default root metastore location linked to an ADLS container.At the same time we are working on a ADLS migration project.All of the tables and files (everything we use for day to day business related) have moved to the new ADLS so ...
- 207 Views
- 2 replies
- 1 kudos
Latest Reply
- 1 kudos
Hi @yawningbrain, You can't repoint the metastore root in place, so the supported path is to give every catalog its own managed storage on the new ADLS and then remove the metastore root. Here is the sequence with the relevant docs. Step 1 - Give you...
- 1 kudos
- 32 Views
- 0 replies
- 0 kudos
Managing "Secret" Injection in DABs across Dev/Stage/Prod
Hi everyone, I am fully moving our team to Databricks Asset Bundles (DABs) for CI/CD, but I’m struggling with the "Secret" management pattern.How are you handling the injection of secrets (like API keys for ingestion) into DABs without hardcoding any...
- 32 Views
- 0 replies
- 0 kudos
- 30 Views
- 0 replies
- 0 kudos
Architectural Pattern for PII: "Redact at Ingestion" vs. "Dynamic Masking at Consumption
Hi everyone, we are defining our PII strategy in Unity Catalog. We are split on whether to: A) Redact/Hash PII at the Silver layer (permanent change), or B) Keep PII in Silver and use Dynamic Data Masking at the Gold/View layer.Does "Redact at Inges...
- 30 Views
- 0 replies
- 0 kudos
- 32 Views
- 0 replies
- 0 kudos
Managing Row-Level Security (RLS) vs. Views: Performance impacts at scale
Hi everyone, we are designing our security layer in Unity Catalog. We are debating between using Row-Level Security (RLS) predicates versus standard SQL Views for masking PII data.When applying complex RLS predicates to tables with >100M rows, have ...
- 32 Views
- 0 replies
- 0 kudos
- 32 Views
- 0 replies
- 0 kudos
Scaling Delta Sharing across Non-Databricks Recipients: Handling Credential Rotation and Lifecycle
Hi everyone,Hi everyone, I’m architecting a data-sharing program using Delta Sharing for external partners who do not have a Databricks workspace.How are others automating the recipient credential lifecycle? Do you have an automated process for rotat...
- 32 Views
- 0 replies
- 0 kudos
- 103 Views
- 1 replies
- 0 kudos
Migrate Azure Databricks between cross tenant
How to migrate an Azure Databricks to a different Azure tenant?Does DAB is the best method if a databricks workspace is freshly built on top of just migrating the the workspace resources only like notebook, jobs, cluster, policies, permissions etc? P...
- 103 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
A fresh workspace makes DAB a reasonable option, especially since you mainly need to migrate notebooks, jobs, policies, clusters, and permissions.
- 0 kudos
- 922 Views
- 9 replies
- 2 kudos
Resolved! Using autoloader with multiple object types in load path
My data source is going to generate csv files for multiple objects all into the same directory that I need to load from. The files will have names in the format along the lines of <objecttype>_YYYY_MM_DD_guid.csv.gz. Each objecttype will have it's ow...
- 922 Views
- 9 replies
- 2 kudos
Latest Reply
- 2 kudos
A single Auto Loader stream with filename-based routing looks much cleaner here, especially since new object types can appear without updating the pipeline.
- 2 kudos
- 131 Views
- 2 replies
- 2 kudos
Liquid Clustering vs Z-Order vs Partitioning
Liquid clustering: what it actually replaced, and the caveats worth knowingI see a lot of questions here that boil down to "should I partition, Z-order, or use liquid clustering." Posting how I reason about it, partly to help and partly because I'd ...
- 131 Views
- 2 replies
- 2 kudos
Latest Reply
- 2 kudos
This lines up well with what Databricks' own docs and the 2026 refresh of their partitioning guidance say — a few things worth adding as confirmation/extra data points rather than pushback:On the key-count cap: confirmed at 4 clustering keys max. Wor...
- 2 kudos
- 66 Views
- 0 replies
- 0 kudos
DQX - Everything The Docs Don’t Say Yet
DQX Studio already has its own documentation for the rule editor, the approval workflow, the scheduler. This is what running it against real pipelines, real approvers, and a growing quarantine table made us add on top of it — and around it. It'll be ...
- 66 Views
- 0 replies
- 0 kudos
- 63 Views
- 0 replies
- 0 kudos
Unmasking the Connected Consumer
Behind every clean “Golden Record” is a messy web of shared emails, phones, and legacy IDs. Here’s how we turned that web into an interactive, cluster-aware graph a data steward can actually work with — built on Streamlit, NetworkX, and a Databricks ...
- 63 Views
- 0 replies
- 0 kudos
- 35145 Views
- 10 replies
- 8 kudos
Unity Catalog-Error Creating Table ErrorClass=INVALID_STATE] Failed to access cloud storage: AbfsRestOperationException
I have managed to create my catalog and schema on unity catalog. Added Storage Blob Contributor role to the unity catalog metastore in ADLS for the Access Connector for Azure Databricks.however, when trying to create a table i received this error:Err...
- 35145 Views
- 10 replies
- 8 kudos
Latest Reply
- 8 kudos
I just faced same issue and was navigating to this thread, and I proactively enabled this option in the metastore settings: that resolved the issue.Hope this helps someone.
- 8 kudos
Announcements
| User | Count |
|---|---|
| 2293 | |
| 1425 | |
| 1253 | |
| 1187 | |
| 961 |
Latest from our Blog
Tutorial: Databricks Genie for Data Engineers and Data Scientists
Note: This blog post is a summary of a Databricks Genie step-by-step tutorial. The full tutorial and code are available for free on GitHub. Tutorial: Databricks Genie for Data Engineers and Data Scien...
Beyond the Dashboard: How Transferz Built a Truly Data-Driven Company Culture With AI/BI Genie
Transferz, a fast-growing travel tech platform, eliminated its data team bottleneck by deploying AI/BI Genie — achieving 89,5% adoption across business users achieving 60% operational improvements.Th...
5 Ways to Enhance Databricks AI/BI Dashboard Tables with HTML
Organizations use AI/BI dashboard tables to review accounts, catch problems, and decide what to do next. Yet a plain table can hold all the right data and still be hard to read. One way to improve the...
Tutorial: Transform your Lakeflow Connect ad data into visual and conversational analytics
Introduction Traditionally, unifying requires building brittle custom integrations for every single source. Lakeflow Connect resolves this with native, fully managed connectors for a full suite of mar...
