Welcome to the family 2,800 new members.And counting. DAIS 2026 brought a wave of new faces to the Databricks Community. If you just joined, welcome. This post is for you: what just happened, what Community is, and what you can do here. What...
- 656 Views
- 1 replies
- 4 kudos
Thursday
Grow Your Skills and Earn Rewards! Mark your calendar: June 15 – July 6, 2026 Join us for a three-week event dedicated to learning, upskilling, and advancing your career in data engineering, analytics, machine learning, and generative AI. Whethe...
- 134906 Views
- 295 replies
- 109 kudos
05-14-2026
Join Our Business Impact Accelerator Sessions! We are thrilled to invite you to the Business Impact Accelerator, a specialized single-session program designed specifically for business professionals, executives, and non-technical teams. If yo...
- 1607 Views
- 1 replies
- 3 kudos
3 weeks ago
Learn how the lakebase architecture decouples compute from storage, so you can build, ship and run agents and apps faster. Databricks has introduced a new category of serverless Postgres database built to integrate seamlessly with your lakehouse sta...
- 1161 Views
- 0 replies
- 0 kudos
2 weeks ago
Our Community Champion Program celebrates members who go above and beyond to share knowledge, support fellow practitioners, and help make the Databricks Community a valuable place to learn and grow. Each month, we recognize individuals whose expertis...
- 315 Views
- 5 replies
- 5 kudos
Friday
Brand imagery plays an important role in how retail companies express their identity and connect with customers. The Creating Brand-Aligned Images Using Generative AI Solution Accelerator shows how organizations can use generative AI to interpret ele...
- 242 Views
- 0 replies
- 1 kudos
Wednesday
The Weekly Digest • June 15 – June 21 Community Pulse DAIS was electric – and the community didn't hold back.Fair warning: this edition came out extra vibrant, DAIS had that effect on us. This Week's Community Stars A warm welcome to our newes...
- 418 Views
- 3 replies
- 4 kudos
Wednesday
Community Activity
- 40 Views
- 0 replies
- 0 kudos
Databricks Visual Data Prep with Star Schema Data Modelling Technique
In this video, you'll learn how to use Databricks Visual Data Prep to clean, transform, and enrich data using a no-code, AI-assisted visual interface for building production-ready data pipelines. Whether you're a data engineer, data analyst, or just...
- 40 Views
- 0 replies
- 0 kudos
- 55 Views
- 0 replies
- 0 kudos
Five Unity Catalog ABAC Updates Worth Paying Attention To
ABAC in Unity Catalog already gave teams a better way to apply row filters and column masks at scale. Instead of rebuilding the same access logic table by table, teams can classify data with governed tags and let policies apply automatically across a...
- 55 Views
- 0 replies
- 0 kudos
- 165 Views
- 0 replies
- 0 kudos
How would you design a Spark pipeline to process billions of records efficiently?
Interview Question:Many people start with the row count.I would start with the architecture.Billions of records are not new in enterprise data engineering. The real challenge is designing a pipeline that runs predictably, efficiently, and within SLA....
- 165 Views
- 0 replies
- 0 kudos
- 90 Views
- 0 replies
- 0 kudos
MCP on Databricks
Here’s What Nobody Tells You.A hands-on look at Genie MCP and DBSQL MCP — what works, what doesn’t, and when to skip MCP entirely.Let me start with something most MCP content skips. MCP servers don’t just expose tools. They expose three things — tool...
- 90 Views
- 0 replies
- 0 kudos
- 6856 Views
- 3 replies
- 2 kudos
Deprecation of DBFS?
I am using Azure Databricks DBFS. I have stored some python scripts, notebooks in folders in DBFS root and have mounted ADLS Gen2 to DBFS. Currently I do not have Unity Catalog enabled and this was setup 3-4 years ago. I came across this article http...
- 6856 Views
- 3 replies
- 2 kudos
Latest Reply
- 2 kudos
In current state, when DBFS is disabled in a workspace, both the root filestore and mount paths become inaccessible.What is deprecated are:DBFS root storage the workspace‑provisioned filestore - /dbfs/FileStore/, /dbfs/root/ etc.DBFS mounts (/dbfs/mn...
- 2 kudos
- 372 Views
- 7 replies
- 5 kudos
Resolved! Data Dictionary and Unity Catalog: Best Practices in a Medallion Architecture
Hi everyone,I'm currently working on implementing a data dictionary, and I'm struggling to understand what the best approach is when using a Medallion architecture in Databricks.My company is migrating its data to Databricks so that users can consume...
- 372 Views
- 7 replies
- 5 kudos
Latest Reply
- 5 kudos
Hi, This is a challenging area to get right, especially with so many tables and schemas. Unity Catalog is definitely a good bet for storing this metadata. There are a few approaches that you could take to the data definitions part, and it depends on ...
- 5 kudos
- 79 Views
- 1 replies
- 0 kudos
Code on cluster runs idefinitely
Hello.Ive created a custom cluster - m4.large. When i try to execute some code in this cluster the behaviour is as follows:- Cluster starts, have running status- I run code, for example print("Hello")- Code runs indefinitely- I click interrupt, it st...
- 79 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
The fact that print("Hello") eventually works but SELECT 1 never completes suggests the cluster may be running but not fully initialized for Spark workloads.A few things I’d check first:Cluster Event Log for any provisioning or startup errors.Spark U...
- 0 kudos
- 443 Views
- 4 replies
- 1 kudos
AI Is Becoming a Data Problem Before It Becomes a Model Problem
The conversation around AI often starts with models. In reality, long-term value comes from how well data pipelines, governance, and workflows work together.The teams moving fastest seem to be simplifying their stack instead of expanding it.What tren...
- 443 Views
- 4 replies
- 1 kudos
Latest Reply
- 1 kudos
Coming from a statistics background, this makes a lot of sense to me. Before any model, we always spend most of the time just cleaning and validating data. the actual modeling is usually the smaller part.I think that's why the medallion architecture ...
- 1 kudos
- 315 Views
- 5 replies
- 5 kudos
Databricks Community Champion - June 2026 - Amira Bedhiafi
Our Community Champion Program celebrates members who go above and beyond to share knowledge, support fellow practitioners, and help make the Databricks Community a valuable place to learn and grow. Each month, we recognize individuals whose expertis...
- 315 Views
- 5 replies
- 5 kudos
- 106 Views
- 1 replies
- 0 kudos
DAB best practices suggestion
We're currently setting up Databricks Asset Bundles (DAB) with a CI/CD pipeline using Azure DevOps.Our planned development workflow is as follows:Main branch → Developer creates a feature branch → Implement changes → Create a Pull Request → Senior de...
- 106 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
You can create Databricks Asset Bundles that are decoupled by domain, managed via multi target declarations within configuration and also driven by immutable, versioned artifacts stored securely within Unity Catalog Volumes. You can rely on explicit ...
- 0 kudos
- 176 Views
- 3 replies
- 0 kudos
Bundle deployment overwrites artifacts while jobs are running - best practices?
Hi everyone,I'm using #Declarative Automation Bundles (DAB) to deploy data pipelines, and I've run into an issue with concurrent job runs and deploymentWhat happened:I started a job that depends on a wheel file built by the bundle (timestamped artifa...
- 176 Views
- 3 replies
- 0 kudos
Latest Reply
- 0 kudos
Do you have source_linked_deployment set as false? That's probably causing it.
- 0 kudos
- 516 Views
- 14 replies
- 5 kudos
Resolved! PySpark AnalysisException: Ambiguous reference to field t when parsing nested JSON
I'm working on a personal data engineering project using Kafka, Spark Structured Streaming, and Docker.The application consumes two Kafka topics that originate from an external market-data websocket source:a trade streama candlestick (kline/OHLCV) st...
- 516 Views
- 14 replies
- 5 kudos
Latest Reply
- 5 kudos
Spark Structured Streaming writes to file sinks and generally it uses a phased commit by writing temporary files to the output directory followed by writing metadata with references and a final commit by moving/renaming temp files to final names. You...
- 5 kudos
- 205 Views
- 0 replies
- 0 kudos
LTAP: What Databricks New Transactional-Analytical Architecture Means for Data Engineers
For years, enterprise data architecture has followed a familiar pattern.An application writes customer orders, account updates, inventory changes, or transactions into an operational database.Then data engineering takes over.We capture changes throug...
- 205 Views
- 0 replies
- 0 kudos
- 117 Views
- 1 replies
- 0 kudos
Need Labs Access for Data Engineering Learning Plan
I am currently going through the Data Engineering Learning Plan. As I progress through the course, I noticed that it refers to labs, but I'm not sure where I can find or access them.It would be very helpful if someone could guide me on how to access ...
- 117 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
The labs referenced in the Data Engineer Learning Plan aren’t included in the free learning plan itself.Those hands-on labs are available through the Databricks Academy Labs subscription, which provides access to the lab-enabled versions of the cours...
- 0 kudos
- 105 Views
- 0 replies
- 0 kudos
runtime CI/CD
Not only is code CI/CD possible, but runtime CI/CD is now possible thanks to #databricks docker. more about it: https://databrickster.medium.com/databricks-docker-from-runtime-ci-cd-to-compliance-1479cf6cdf8d https://www.sunnydata.ai/blog/databricks-...
- 105 Views
- 0 replies
- 0 kudos
Announcements
| User | Count |
|---|---|
| 2276 | |
| 1323 | |
| 1232 | |
| 1182 | |
| 960 |
| User | Count |
|---|---|
| 442 | |
| 286 | |
| 220 | |
| 63 | |
| 51 |
Latest from our Blog
Features in Motion: Three Patterns for Real-Time ML in Databricks Lakebase
Summary In production machine learning systems (like real-time recommenders or fraud detection), models operate under a strict latency budget (typically milliseconds end-to-end). Within this window, t...
Incremental REPLACE WHERE Flows Brings Targeted Refreshes to SDP and DBSQL
Lakeflow Spark Declarative Pipelines (SDP) has long supported two ways to maintain a table: a streaming flow for incremental ingestion and a materialized view for a fully derived result that is always...
One question, two data realities: supervisor agents across Kafka and the lakehouse
The setup every operations analyst knows A procurement lead asks what sounds like a simple question:"Of the orders that came in over the last hour, which ones are from customers who are close to their...
Triggering Databricks Jobs from Celonis Action Flows — A Step-by-Step Guide
Turn process intelligence into automated action: connect Celonis directly to Databricks. Why Connect Celonis Action Flows to Databricks? Celonis shows you where your business processes break down — wh...
