We are excited to announce that the Databricks Community has been nominated for Best Customer Support Community in the CMX Community Industry Awards! This nomination is a testament to the incredible collaboration and support within our community. Fr...
- 85 Views
- 0 replies
- 2 kudos
Monday
Join us for the return of the Virtual Learning Festival! Mark your calendars from 9 April - 30 April 2025! Upskill across data engineering, data analysis, machine learning, and generative AI. Join the thousands who have elevated their career with...
- 3583 Views
- 16 replies
- 6 kudos
2 weeks ago
Learn the foundational principles of the lakehouse architecture with three short self-paced videos The architecture of your data platform directly affects the reliability, performance and scalability of your data and AI initiatives. The lakehouse arc...
- 747 Views
- 0 replies
- 2 kudos
2 weeks ago
Be part of a global movement! Connect with 22,000 data enthusiasts at 700+ sessions, keynotes, and hands-on training at this year’s Data + AI Summit. Whether you’re into data intelligence, governance, AI, or data warehousing, this is your chance to l...
- 1878 Views
- 0 replies
- 3 kudos
a month ago
The Databricks Developer Relations team is excited to share that we are hitting the road and bringing a series of community meetups to you called DevConnect! These events will be hosted around the globe and tailored specifically for Data engineers. T...
- 3851 Views
- 2 replies
- 1 kudos
a month ago
Meet our Databricks Community Champion, Stefan Koch! From a career in law enforcement to becoming a Principal Solution Architect at b.telligent, Stefan’s journey is nothing short of inspiring. Read on to learn about his experience, favorite Databri...
- 1189 Views
- 6 replies
- 6 kudos
2 weeks ago
Community Activity
- 6594 Views
- 10 replies
- 11 kudos
Can we add tags to Unity Catalog through Terraform?
We use Terraform to manage most of our infrastructure, and I would like to extend this to Unity Catalog. However, we are extensive users of tagging to categorize our datasets, and the only programmatic method I can find for adding tags is to use SQL ...
- 6594 Views
- 10 replies
- 11 kudos
Latest Reply
- 11 kudos
It seems very frustrating to many of the users and not to say probably the big clients, waiting over a year for this feature. Anyone from Databricks that could raise this internally to get it done-done would be very appreciated.
- 11 kudos
- 29 Views
- 1 replies
- 0 kudos
Query related to Job cluster verses All Purpose Cluster
Currently if run the Notebook with Job Cluster it will show status as Pending for 2 minutes but All Purpose Cluster shows only 6seconds as pending and completes the run quickly. How to improve startup time of Job Cluster to mmatch with startup time ...
- 29 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi @Dileep_Karanki Thank you for reaching out and for your question. Here is a link for documentation about the available cluster types: https://docs.databricks.com/aws/en/compute#types-of-compute All-Purpose or Interactive Clusters: These are flex...
- 0 kudos
- 9 Views
- 1 replies
- 0 kudos
Schedule & Triggers
Hi guys, having some issue when I use Schedule and trigger in workflow when file arrives condition, if a place a file with same name it won't trigger is there any configuration I need to do in databricks? thanks
- 9 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi @Leog Thank you for reaching out and for your sending your question. Per the documentation: https://docs.databricks.com/gcp/en/jobs/file-arrival-triggers Only new files trigger runs. Overwriting an existing file with a file of the same name does ...
- 0 kudos
- 15 Views
- 0 replies
- 0 kudos
Assigning Dedicated (SINGLE_USER) ML Clusters to a Group in Databricks
I'm working with Databricks Runtime ML and have configured a cluster in Dedicated access mode (formerly SINGLE_USER). The documentation indicates that a compute resource with Dedicated access can be assigned to a group, allowing user permissions to a...
- 15 Views
- 0 replies
- 0 kudos
- 8 Views
- 0 replies
- 0 kudos
Asset Bundles Email/Notifications Prod ONly
Howdy, I've got a job 'job1' and my dev/stg/prod target in my databricks.yamlCurrently, I have this configurationfor my job:email_notifications: on_success: - me@myorg.com on_failure: - me@myorg.comwebhook_notifications: on_failure: - id: ${var.w...
- 8 Views
- 0 replies
- 0 kudos
- 273 Views
- 4 replies
- 0 kudos
Self-joins are blocked on remote tables
In our production databricks workflow, we have been getting this error since yesterday in one of the steps:org.apache.spark.SparkException: Self-joins are blocked on remote tablesWe haven't changed our workflow or made any configurations for the data...
- 273 Views
- 4 replies
- 0 kudos
Latest Reply
- 0 kudos
@chris_y_1e take a look at your cluster config to see if you're tripping up on the same condition I was. See my comment, above.
- 0 kudos
- 9666 Views
- 14 replies
- 11 kudos
- 9666 Views
- 14 replies
- 11 kudos
Latest Reply
- 11 kudos
I'm also interested in letting the auth for notebooks/dashboard embedding been delegated to the app, is there any progress been made in that capability? And maybe a stupid question, how do I reach out to my account team?
- 11 kudos
- 739 Views
- 7 replies
- 2 kudos
Resolved! When will DB release runtime with Scala 2.13
When will DB release runtime with Scala 2.13? Thx
- 739 Views
- 7 replies
- 2 kudos
Latest Reply
- 2 kudos
Spark 4 is coming and it's only going to support Scala 2.13, so a Databricks Runtime with Spark 3.5.x and Scala 2.13 should be released soonish.
- 2 kudos
- 902 Views
- 4 replies
- 0 kudos
LangGraph MemorySaver checkpointer usage with MLflow
Hi everyone.I am working on a graph that utilizes the MemorySaver class to incorporate short-term memory. This will enable me to maintain a multi-turn conversation with the user by storing the chat history.I am using the MLflow "models from code" fea...
- 902 Views
- 4 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi @moemedina. No, I didn't.I'm considering using ChatModel/ChatAgent class to wrap the graph and be able to move on. However, the MLflow documentation is still referring to ChatModel where Chat Agent is the latest recommendation:MLflow ChatModel Doc...
- 0 kudos
- 133 Views
- 4 replies
- 0 kudos
Unable to connect to the cluster
Container launch failure:An unexpected error was encountered while launching containers on worker instances for the cluster. Please retry and contact Databricks if the problem persists.Instance ID: i-02aadc9a7c3532af0Internal error message: Failed to...
- 133 Views
- 4 replies
- 0 kudos
- 29 Views
- 0 replies
- 0 kudos
DLT Streaming With Watermark fails, suggesting I should add watermarks
Hi all,I have the following Problem: I have two streaming tables containing time-series measurements from different sensor data, each feed by multiple sensors. (Imagine: Multiple Temperature Sensors for the first table, and multiple humidity sensors ...
- 29 Views
- 0 replies
- 0 kudos
- 9415 Views
- 21 replies
- 1 kudos
Resolved! My Databrick exam got suspended just for coming closer to laptop screen to read question and options
Hi team,My Databricks Certified Data Engineer Associate exam got suspended within 10 minutes.I had also shown my exam room to the proctor. My exam got suspended due to eye movement. I was not moving my eyes away from laptop screen. It's hard to focus...
- 9415 Views
- 21 replies
- 1 kudos
- 50 Views
- 1 replies
- 0 kudos
Subject: Regarding Ticket 00634058
Hi Team,I have raised the ticket #00634058 requesting to reschdule Databricks Data Engineer Associate Exam. I haven't got response on it yet. Can anybody please check and help me to reschedule the exam asap.Email Id: ashishlimaye93@gmail.com
- 50 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Hello @ashishlimaye ,We are sorry to hear that your exam was suspended. Thank you for filing a ticket with our support team. Please allow the support team 24-48 hours for a resolution. In the meantime, you can review the following documentation: Beha...
- 0 kudos
- 28 Views
- 0 replies
- 0 kudos
Databricks Serving Endpoint 400 Error: Model Response Format Issue while langgraph tool calling
Dear Databricks Community,I am seeking assistance with an issue I've encountered while deploying a model on Databricks. When invoking the serving endpoint, I receive the following error message sometimes:400 Client Error: Error: Model response did no...
- 28 Views
- 0 replies
- 0 kudos
- 48 Views
- 1 replies
- 0 kudos
How to Query All the users who have access to a databricks workspace?
Hi There,I'm new to Databricks and we currently have a lot of users among different groups having access to a databricks workspace. I would like to know how I could query the users, groups and Entitlements of each groups using SQL or the API. Incase ...
- 48 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
To query all users who have access to a Databricks workspace, you can follow these steps:1. Check Workspace Users via Admin ConsoleIf you are a workspace admin, navigate to the Admin Console in the Databricks UI. Under the "Users" tab, you can view a...
- 0 kudos
Announcements
| User | Count |
|---|---|
| 106 | |
| 43 | |
| 22 | |
| 14 | |
| 13 |
Latest from our Blog
How to Assess and Adopt New Technologies Using Data-Driven Proof of Concept
In this article, I share a #data-driven method for assessing and adopting new technologies and steps for creating or evaluating a strong proposal. This includes using statistical measures to evaluate ...
[PARTNER BLOG] Introducing Databricks Native Master Data Management (MDM) - Entity Resolution
LakeFusion’s MDM solution delivers a single source of truth by leveraging advanced entity resolution and deduplication algorithms. By unifying fragmented data across systems, we ensure accurate, consi...
💡 ML Training Tip Of The Week #5: Enable fine-grained data frequency in AutoML forecast
Customers with high-frequency time series data often need forecasts at a finer granularity based on their data sources. For example, an energy supply company may receive energy demand data every 15 mi...
💡 ML Training Tip Of The Week #4: Speed up your ML workload with one toggle
Machine learning workload can be time-consuming due to training data processing such as ETL and feature engineering and iterative model training steps. Recent ML Runtime releases have made both faster...
