Grow Your Skills and Earn Rewards! Mark your calendar: January 09 – January 30, 2026 Join us for a three-week event dedicated to learning, upskilling, and advancing your career in data engineering, analytics, machine learning, and generative AI. ...
- 119306 Views
- 379 replies
- 133 kudos
12-09-2025
June 15–18, 2026 Are you building the future with data and AI? Then this is your moment. The Call for Proposals for Data + AI Summit 2026 is officially open, and we want to hear from builders, practitioners, and innovators across the data and AI com...
- 7654 Views
- 4 replies
- 6 kudos
12-15-2025
How to Register & Prepare If you're interested in advancing your skills with Databricks through a Specialist Session, here's a clear guide on how to register and what free courses you can take to prepare effectively. How to Begin Your Learning Path S...
- 3980 Views
- 2 replies
- 9 kudos
10-02-2025
Strengthening cybersecurity isn’t a one-time project — it’s an ongoing process of adapting to new and complex threats. The focus is on modernizing your data infrastructure and analytics capabilities to make security operations smarter, faster, and mo...
- 903 Views
- 1 replies
- 2 kudos
2 weeks ago
2025 wasn’t just another year on the Databricks Community.It was a year of showing up, helping each other, and building trust. Watch the 2025 Community Wrap-Up This video is a small reflection of that journey.It highlights the key contributors of 2...
- 1344 Views
- 13 replies
- 20 kudos
2 weeks ago
Another strong week of engagement, learning, and problem-solving across the community.Let’s take a look at the highlights Standout Contributors These members stepped in with sharp insights, thoughtful follow-ups, and the kind of curiosity that ke...
- 242 Views
- 0 replies
- 0 kudos
Tuesday
Community Activity
- 741 Views
- 4 replies
- 5 kudos
Resolved! switching to Databrick from Ab Initio (an old ETL software)- NEED ADVICE
All courses in market and on youtube as per my knowledge for databrick is outdated as those courses are for community edition. there is no new course for free edition of databrick. i am a working profession and i do not get much time. do you guys kno...
- 741 Views
- 4 replies
- 5 kudos
Latest Reply
- 5 kudos
@pop_smoke What were the arguments that swayed your organization to swtich to Databricks from Ab Initiio?Pure cost basis?
- 5 kudos
- 16 Views
- 0 replies
- 1 kudos
Databricks Assistant
Databricks Assistant can also be used in databricks documentation without login to #databricks. Read and watch databricks news on:https://databrickster.medium.com/databricks-news-2026-week-2-5-january-2026-to-11-january-2026-0bfc6c592051
- 16 Views
- 0 replies
- 1 kudos
- 20 Views
- 1 replies
- 0 kudos
al purpose databricks cluster disappear
Hi, i can see sometime the cluster get disappeared even though it was created some time back using cluster pipeline,what could be the reason to disappear.we can recreate cluster but wanted to know the reason why this cluster get disappeared.thanks!Ve...
- 20 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi @Ved88 ,30 days after a compute is terminated, it is permanently deleted. To keep an all-purpose compute configuration after a compute has been terminated for more than 30 days, an administrator can pin the compute. Up to 100 compute resources can...
- 0 kudos
- 26 Views
- 0 replies
- 0 kudos
Publish a Technical Blog
Sir/Madam,I hope you are doing well. I would like to contribute and publish a technical blog on the Databricks Community portal. However, I am not sure about the process and appropriate platform. I would appreciate your guidance on where to start, th...
- 26 Views
- 0 replies
- 0 kudos
- 38 Views
- 2 replies
- 0 kudos
Same Delta Table, Different Behavior: Dev vs Prod Workspace in Databricks
I recently ran into an interesting Databricks behavior while implementing a row-count comparison using Delta Time Travel (VERSION AS OF).Platform: AzureScenario:Same Unity CatalogSame fully qualified tableSame table ID, location, and Delta formatYet ...
- 38 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
This could happen if the Delta table in production is a streaming table, while the Delta table in development is a regular Delta table. Time travel doesn’t work well with streaming tables and materialized views.
- 0 kudos
- 33 Views
- 0 replies
- 0 kudos
Unity Catalog Schema management
From time to time i read articles such as here which suggest to use a unity catalog schema management tool. All table schema changes should be applied via this tool.Usually SPs (or users) have the "Modify" Permission on tables. This allows to them t...
- 33 Views
- 0 replies
- 0 kudos
- 2354 Views
- 3 replies
- 4 kudos
Getting Databricks Generative AI Engineer Associate (and What I Learned)
I just earned my Databricks Certified Generative AI Engineer Associate Certification, and in this post, I’m sharing the key tips, resources, and personal insights that helped me succeed. My certiticate from DatabricksNavigation:PreludeAbout the certR...
- 2354 Views
- 3 replies
- 4 kudos
Latest Reply
- 4 kudos
Hi @devipriya Contrast and thanks' for shearing use full key tips, resources, and personal insights. it's really helpful for others...
- 4 kudos
- 23 Views
- 0 replies
- 0 kudos
Query: INR Payment Option & Certification Cost Considerations
Hello Team,I had a question regarding the payment options and overall cost for Databricks certifications, particularly from an India-based learner perspective.Is there currently an option to pay the certification fees in INR? If not, I wanted to high...
- 23 Views
- 0 replies
- 0 kudos
- 27 Views
- 0 replies
- 1 kudos
Security considerations and model customization options for AgentBricks AI Agents and LLM Judges
Hello Databricks Team,We are currently evaluating AgentBricks AI Agents (for example, Knowledge Assistant and Multi-Agent Supervisor) and would like to better understand the security and model customization aspects.Data security & privacyWhat data (u...
- 27 Views
- 0 replies
- 1 kudos
- 35 Views
- 0 replies
- 0 kudos
Automate Resource Creation using Databricks SDK for Python in VS Code
In this video article, I demonstrated how to automate resource creation in Azure Databricks using the Databricks SDK for Python. If you've ever found yourself manually creating catalogs, schemas, and volumes in the Databricks UI repeatedly, this vid...
- 35 Views
- 0 replies
- 0 kudos
- 78 Views
- 0 replies
- 0 kudos
Let's talk about Data Governance
As we move into 2026, the mandate for modern data and AI is clear. The industry has pivoted from evaluating isolated features to building foundational futures. Since the 2025 Data + AI Summit, I have consulted with numerous organizations across a ran...
- 78 Views
- 0 replies
- 0 kudos
- 91 Views
- 2 replies
- 2 kudos
Python DataSource API utilities/ Import Fails in Spark Declarative Pipeline
TLDR - UDFs work fine when imported from `utilities/` folder in DLT pipelines, but custom Python DataSource APIs fail with ModuleNotFoundError: No module named 'utilities'` during serialization. Only inline definitions work. Need reusable DataSource ...
- 91 Views
- 2 replies
- 2 kudos
Latest Reply
- 2 kudos
@emma_s Thank you for the guidance! The wheel package approach worked perfectly.I also tried putting the .py directly in but it did not work/Workspace/Libraries/custom_datasource.py
- 2 kudos
- 92 Views
- 2 replies
- 2 kudos
Lakeflow Spark Declarative Pipeline
How we can run a SDP pipeline in parallel manner with dynamic parameter parsing on pipeline level. How we can consume job level parameter in Pipeline. If similar name parameters are defined in pipeline level then job level parameters are getting over...
- 92 Views
- 2 replies
- 2 kudos
Latest Reply
- 2 kudos
Just FYI, as of Jan 16th (the time I'm writing this answer), SDP and Delta Lake in their OSS versions don't work together yet.SDP is part of Apache Spark 4.1, but Delta Lake does not support it at the moment. It's coming. No idea when it's gonna be a...
- 2 kudos
- 92 Views
- 3 replies
- 4 kudos
Resolved! Is Auto Loader open source now in Apache 4.1 SDP?
With Spark Declarative Pipelines (SDP) being open source now, does this mean that the Databricks Auto Loader functionality is also open source? Is it called something else? If not, how does the open-source version handle incremental data processing a...
- 92 Views
- 3 replies
- 4 kudos
Latest Reply
- 4 kudos
Hi @ChristianRRL ,No, autoloader is propriety to Databricks. It's not open sourced. Open source version of SDP uses spark structured streaming for incremental processing. Keep in mind that Auto Loader is basically just Spark streaming under the hood ...
- 4 kudos
- 43 Views
- 0 replies
- 0 kudos
External to Managed
We are looking to migrate to managed tables using ALTER TABLE fq_table_name SET MANAGED During migration process we need to have ability to switch between external & managed tables & vice versa.UNSET MANAGED works for 14 days. But I'm unable to just ...
- 43 Views
- 0 replies
- 0 kudos
Announcements
| User | Count |
|---|---|
| 2246 | |
| 1227 | |
| 1087 | |
| 1032 | |
| 950 |
Latest from our Blog
10 things you no longer need to think about (but if you are, here's how to fix it)
Databricks just released a button to fix all legacy issues in your workspace. Actually, they didn't. But the fact that you felt a momentary surge of hope—or a cynical eye-roll—proves exactly where you...
From Traces to Evaluation: An Autonomous MLflow Agent Harness for AI AgentOps
The Gap in Agent Development Databricks has made it straightforward to deploy AI agents—Model Serving endpoints, automatic MLflow tracing, Unity Catalog integration. But there's a gap between "deploye...
Combining Agent Bricks and ai_parse_document for Smarter Document Workflows
Building a Research Paper Curator for Knowledge Assistants Document-heavy workflows such as research analysis, contract review, and support ticket processing, are a natural fit for AI agents. Building...
Declarative Pipelines meets foreachBatch: Custom Streaming for Advanced Pipelines
If this is the first time you’re hearing about Sinks in Lakeflow Declarative Pipelines (LDP), we highly recommend reading Introducing the SDP Sink API, which explores the recently launched Sinks API, ...
