Grow Your Skills and Earn Rewards! Mark your calendar: January 09 – January 30, 2026 Join us for a three-week event dedicated to learning, upskilling, and advancing your career in data engineering, analytics, machine learning, and generative AI. ...
- 115170 Views
- 367 replies
- 133 kudos
12-09-2025
June 15–18, 2026 Are you building the future with data and AI? Then this is your moment. The Call for Proposals for Data + AI Summit 2026 is officially open, and we want to hear from builders, practitioners, and innovators across the data and AI com...
- 7525 Views
- 4 replies
- 6 kudos
12-15-2025
How to Register & Prepare If you're interested in advancing your skills with Databricks through a Specialist Session, here's a clear guide on how to register and what free courses you can take to prepare effectively. How to Begin Your Learning Path S...
- 3960 Views
- 2 replies
- 9 kudos
10-02-2025
Strengthening cybersecurity isn’t a one-time project — it’s an ongoing process of adapting to new and complex threats. The focus is on modernizing your data infrastructure and analytics capabilities to make security operations smarter, faster, and mo...
- 853 Views
- 1 replies
- 2 kudos
2 weeks ago
2025 wasn’t just another year on the Databricks Community.It was a year of showing up, helping each other, and building trust. Watch the 2025 Community Wrap-Up This video is a small reflection of that journey.It highlights the key contributors of 2...
- 1300 Views
- 13 replies
- 20 kudos
a week ago
Another strong week of engagement, learning, and problem-solving across the community.Let’s take a look at the highlights Standout Contributors These members stepped in with sharp insights, thoughtful follow-ups, and the kind of curiosity that ke...
- 212 Views
- 0 replies
- 0 kudos
Tuesday
Community Activity
- 12 Views
- 0 replies
- 0 kudos
Let's talk about Data Governance
As we move into 2026, the mandate for modern data and AI is clear. The industry has pivoted from evaluating isolated features to building foundational futures. Since the 2025 Data + AI Summit, I have consulted with numerous organizations across a ran...
- 12 Views
- 0 replies
- 0 kudos
- 67 Views
- 2 replies
- 1 kudos
Python DataSource API utilities/ Import Fails in Spark Declarative Pipeline
TLDR - UDFs work fine when imported from `utilities/` folder in DLT pipelines, but custom Python DataSource APIs fail with ModuleNotFoundError: No module named 'utilities'` during serialization. Only inline definitions work. Need reusable DataSource ...
- 67 Views
- 2 replies
- 1 kudos
Latest Reply
- 1 kudos
@emma_s Thank you for the guidance! The wheel package approach worked perfectly.I also tried putting the .py directly in but it did not work/Workspace/Libraries/custom_datasource.py
- 1 kudos
- 75 Views
- 2 replies
- 2 kudos
Lakeflow Spark Declarative Pipeline
How we can run a SDP pipeline in parallel manner with dynamic parameter parsing on pipeline level. How we can consume job level parameter in Pipeline. If similar name parameters are defined in pipeline level then job level parameters are getting over...
- 75 Views
- 2 replies
- 2 kudos
Latest Reply
- 2 kudos
Just FYI, as of Jan 16th (the time I'm writing this answer), SDP and Delta Lake in their OSS versions don't work together yet.SDP is part of Apache Spark 4.1, but Delta Lake does not support it at the moment. It's coming. No idea when it's gonna be a...
- 2 kudos
- 57 Views
- 3 replies
- 2 kudos
Resolved! Is Auto Loader open source now in Apache 4.1 SDP?
With Spark Declarative Pipelines (SDP) being open source now, does this mean that the Databricks Auto Loader functionality is also open source? Is it called something else? If not, how does the open-source version handle incremental data processing a...
- 57 Views
- 3 replies
- 2 kudos
Latest Reply
- 2 kudos
Hi @ChristianRRL ,No, autoloader is propriety to Databricks. It's not open sourced. Open source version of SDP uses spark structured streaming for incremental processing. Keep in mind that Auto Loader is basically just Spark streaming under the hood ...
- 2 kudos
- 25 Views
- 0 replies
- 0 kudos
External to Managed
We are looking to migrate to managed tables using ALTER TABLE fq_table_name SET MANAGED During migration process we need to have ability to switch between external & managed tables & vice versa.UNSET MANAGED works for 14 days. But I'm unable to just ...
- 25 Views
- 0 replies
- 0 kudos
- 71 Views
- 4 replies
- 4 kudos
Case insensitive data
For all it's positives, one of the first general issues we had with databricks was case sensitivity.We have a lot of data specific filters in our codeProblem is, we land and view data from lots of different case insensitive source systems e.g. SQL Se...
- 71 Views
- 4 replies
- 4 kudos
Latest Reply
- 4 kudos
Hi, You can set the default collation at Catalog level or schema level and the tables in the catalog will inherit the collation. This is supported from DBR 17.1 and above.
- 4 kudos
- 45 Views
- 1 replies
- 0 kudos
Databricks free Edition to Azure Connectivity
Hello EveryoneAs a beginner in databricks, I have a question. Can we connect Databricks Free Edition to connect Azure Blob/ Gen 2 storage? I would like to create external tables on files on Azure and Delta lake tables on top of it.Your help is apprec...
- 45 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi @Bkr-dbricks ,According to following topic Free Edition doesn't support external locations.Solved: If use databricks free version not free trail can ... - Databricks Community - 127421
- 0 kudos
- 506 Views
- 1 replies
- 0 kudos
HTML Formatting Issue in Databricks Alerts
Hello everyone,I have recently encountered an issue with HTML formatting in custom templates for Databricks Alerts. Previously, the formatting worked correctly, but now the alerts display raw HTML instead of properly rendered content.For example, an ...
- 506 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
hi @kivaniutenko thanks for reaching out. Databricks alerts still support basic HTML in email templates, but HTML will render correctly only for email destinations and only with simple, allowed tags. Quick things to try Make sure you are using Ale...
- 0 kudos
- 67 Views
- 0 replies
- 2 kudos
DABS: New Plan Version
If you are using a plan to deploy DABS, starting from 0.282, plan_version has been moved to 2. A new plan can have a different structure. Please keep in mind that inconsistencies in DABS versions can break your CI/CD. #databricks I wrote an article a...
- 67 Views
- 0 replies
- 2 kudos
- 181 Views
- 2 replies
- 3 kudos
Resolved! Databricks Serverless Pipelines - Incremental Refresh Doubts
Hello everyone,I would like to clarify some doubts regarding how Databricks Pipelines (DLT) behave when using serverless pipelines with incremental updates.In general, incremental processing is enabled and works as expected. However, I have observed ...
- 181 Views
- 2 replies
- 3 kudos
Latest Reply
- 3 kudos
Hi @Alf01 , thanks for accepting the solution! To keep you updated, the REFRESH POLICY feature, that I mentioned in my post, is out now! It allows manual control of the refresh strategy (AUTO, INCREMENTAL, INCREMENTAL STRICT, FULL), just as you stat...
- 3 kudos
- 74 Views
- 2 replies
- 2 kudos
Resolved! Job Cluster Reuse
Hi, I have a job where a job cluster is reused twice for task A and task C. Between A and C, task B runs for 4 hours on a different interactive cluster. The issue here is that the job cluster doesn't terminate as soon as Task A is completed and sits ...
- 74 Views
- 2 replies
- 2 kudos
Latest Reply
- 2 kudos
Hi @SparkMan ,This is expected behavior with Databricks job cluster reuse unless you change your job/task configuration. Look at following documentation entry:So with your flow you have something like this:Task A (job cluster) → Task B (interactive c...
- 2 kudos
- 3001 Views
- 2 replies
- 0 kudos
Jumpstart Your Data Journey with Databricks Get Started Days!
Are you ready to dive into the world of data engineering and analytics? Join us for Databricks Get Started Days, a half-day virtual event designed to accelerate your learning and equip you with essential Databricks skills! Here’s what makes Get Start...
- 3001 Views
- 2 replies
- 0 kudos
- 102 Views
- 2 replies
- 0 kudos
Parity between Spark.sql and SQL Warehouse for Metric Views & Model Visualization
Hi everyone,I’m exploring the new Databricks Metric Views (Semantic Layer) and have two questions regarding programmatic management and UI visualization.1. Parser Disparity: spark.sql vs. SQL WarehouseI'm noticing that CREATE OR REPLACE VIEW ... WITH...
- 102 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi @smpa011,Metric views work on both SQL warehouse and classic clusters, but the cluster must be running DBR 17.2 or above. The error you are getting is because you may be using DBR version below 17.2 or so. try upgrading your DBR version or use SQL...
- 0 kudos
- 59 Views
- 1 replies
- 1 kudos
Regarding Accelerators
Are there any databricks accelerators to convert the c# and qlikview code to pyspark ? We are using the Open source AI tools to convert now but wondering is there any better way to do the same?Thanks in advance
- 59 Views
- 1 replies
- 1 kudos
Latest Reply
- 1 kudos
Hi @nkrish ,Unfortunately, I don't think so. Available accelerators you can find here:Databricks Solution Accelerators for Data & AI | DatabricksBut I haven't heard anything about accelerator for c# and qlikview specifically.
- 1 kudos
- 63 Views
- 1 replies
- 0 kudos
How to Play or Stream MP4 Videos from Unity Catalog Volumes in Databricks (Flask/Dash)?
Hello Databricks Community,I am working on a Dash dashboard (Python/Flask backend) deployed on Databricks, and I need to play or stream MP4 video files stored in a Unity Catalog Volume. I have tried accessing these files both from a Databricks notebo...
- 63 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
@GergoBo - Since notebooks cannot reach out to the file system to stream, you must embed the video as a Base64 encoded string. I tried below code and it works well in Notebook as it plays the video in the output. import base64from IPython.display imp...
- 0 kudos
Announcements
| User | Count |
|---|---|
| 2246 | |
| 1227 | |
| 1085 | |
| 1031 | |
| 950 |
Latest from our Blog
10 things you no longer need to think about (but if you are, here's how to fix it)
Databricks just released a button to fix all legacy issues in your workspace. Actually, they didn't. But the fact that you felt a momentary surge of hope—or a cynical eye-roll—proves exactly where you...
From Traces to Evaluation: An Autonomous MLflow Agent Harness for AI AgentOps
The Gap in Agent Development Databricks has made it straightforward to deploy AI agents—Model Serving endpoints, automatic MLflow tracing, Unity Catalog integration. But there's a gap between "deploye...
Combining Agent Bricks and ai_parse_document for Smarter Document Workflows
Building a Research Paper Curator for Knowledge Assistants Document-heavy workflows such as research analysis, contract review, and support ticket processing, are a natural fit for AI agents. Building...
Declarative Pipelines meets foreachBatch: Custom Streaming for Advanced Pipelines
If this is the first time you’re hearing about Sinks in Lakeflow Declarative Pipelines (LDP), we highly recommend reading Introducing the SDP Sink API, which explores the recently launched Sinks API, ...
