At this year’s Data + AI Summit, we introduced Databricks One, a new experience that brings the power of data and AI to every employee through a clear and focused user interface. That announcement began our journey to remove barriers that have limite...
- 338 Views
- 0 replies
- 2 kudos
yesterday
We are back with another round of appreciation for members who lit up the Databricks Community this week with their contributions, positivity, and support! Here are the Sparks for this week: nayan_wylde – Took initiative in answering questions with...
- 44 Views
- 0 replies
- 1 kudos
2 hours ago
Learn, Level Up, and Get Rewarded! Save the dates: October 10 – October 31, 2025! We're celebrating learning, growth, and career advancement with an exciting opportunity to level up in data engineering, data analysis, machine learning, and genera...
- 51125 Views
- 67 replies
- 48 kudos
08-07-2025
Why this matters:Customer service teams today face increasing pressure to deliver faster, more accurate, and more personalized responses. But scaling human-led support while maintaining quality is costly and challenging. With the LLMs for Customer Se...
- 708 Views
- 2 replies
- 4 kudos
Wednesday
This week was packed with impactful discussions, fresh content, and helpful solutions across the Databricks Community: Learning From Your Instructor: How Can We Make Training Better? Community Events Databricks DevConnect Mexico City | Sept 22 ...
- 412 Views
- 2 replies
- 5 kudos
Wednesday
Data + AI World Tour continues the momentum from Data + AI Summit, the world’s premier event for the data, analytics and AI community. Now we’re coming to a city near you — with 19 stops across five continents. Find out what’s next in data and AI, an...
- 352 Views
- 8 replies
- 2 kudos
yesterday
Community Activity
- 21 Views
- 2 replies
- 1 kudos
How to fetch spark.addFiles when used multi node cluster
I wanted to share the nearly 12 xml files from dbfs location to executor local path by using sc.addFile and I went to your blog and tweaked my code to form path with file:/// the result of it was it worked when we have only one node but throwed erro...
- 21 Views
- 2 replies
- 1 kudos
Latest Reply
- 1 kudos
Hello @Mahesh_rathi__ , SparkContext.addFile is for shipping small side files to executors, not for creating an input path that you can pass to sc.textFile("file://..."). On a single-node cluster the driver and executor share the same machine, so the...
- 1 kudos
- 3000 Views
- 6 replies
- 5 kudos
Resolved! How to get Lab access provided by Vocareum
Hi TeamI have enrolled in the AI/BI for Data Analyst course. Could you please advise how to get the access for LAB to follow the demo instructions in the courseThanks
- 3000 Views
- 6 replies
- 5 kudos
Latest Reply
- 5 kudos
Hi Advika, I tried to get to the Lab suscription, but the weg page is not working. Could you please give me the correct link to access it?Thank you very much.
- 5 kudos
- 329 Views
- 9 replies
- 12 kudos
Workflows now harder to find old failed runs
Some time in the past few weeks I think there was an update to databricks workflows. Previously you could:run a workflowit failsrepair the workflowclick into the workflowview past runs before that failed via a dropdown bar (like in the screenshot bel...
- 329 Views
- 9 replies
- 12 kudos
Latest Reply
- 12 kudos
I am a Databricks Admin in the workspace that @kenmyers-8451 is having problems in and I am getting the same issue where the retries show up for a brief second but then redirect/refresh and then the retries disappear.This seems to happen when the wor...
- 12 kudos
- 169 Views
- 2 replies
- 0 kudos
Serverless base env setup in Databricks Asset Bundle (DAB)
I am trying to set a base environment for my task (notebook) which is running on serverless, following is the dab yaml i am using when i did bundle deploy -t users, its not throwing any error but its not installing the libraries from the base env, ca...
- 169 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
Your YAML is valid, but the reason the libraries are not being installed is because base_environment_path is not supported for serverless compute. Serverless jobs use a fully managed environment and you can’t override it with a custom base environmen...
- 0 kudos
- 42 Views
- 0 replies
- 1 kudos
Normalised ingestion of Vehicle Telematics
Introduction Internet of Things (IoT) technology has increased real-time transmission, processing, and analysis of sensor data. In the automotive industry, this manifests as vehicles equipped with sensors that monitor various components, enabling som...
- 42 Views
- 0 replies
- 1 kudos
- 159 Views
- 8 replies
- 1 kudos
JDBC driver CPU consumption
Hi,I am using JDBC driver to execute an insert statement with several thousand of rows (~4MB). It takes several seconds to complete and for some reason consumes 1 full CPU core for it.It seems like a lot of the time is spent in this method:com.databr...
- 159 Views
- 8 replies
- 1 kudos
Latest Reply
- 1 kudos
Hi @ivni ,Yes, that method could be CPU intensive. According to driver's docs it removes catalog name from query statement. But it doing this via regex patterns - this is heavy operation from CPU perspective, especially if you have a lot of complex q...
- 1 kudos
- 11 Views
- 0 replies
- 1 kudos
Want to Learn more about all the new Features available in Free Edition - We have you Covered!
Features Community Edition Free Edition Notebooks ML Flow Ingestion Jobs Pipelines Dashboards Genie Semantic Search Model Serving Model Evaluation Agents Unity Catalog Clean Rooms ...
- 11 Views
- 0 replies
- 1 kudos
- 258 Views
- 4 replies
- 9 kudos
From Your Instructor: How Can We Make Training Better?
Hello Databricks Community! As a Databricks instructor, I'm looking to gather feedback on your training experiences to help improve our programs. Whether you've taken official Databricks courses, bootcamps, or certification prep - I'd love to hear f...
- 258 Views
- 4 replies
- 9 kudos
Latest Reply
- 9 kudos
Great feedback @BS_THE_ANALYST. I would like to offer my thoughts on each of your points: 1. Blended learning is fairly new to us as instructors. Our only interaction with the Uplimit Blended Learning format is one hour per week during class sessions...
- 9 kudos
- 44 Views
- 0 replies
- 1 kudos
🌟 Community Sparks of the Week | September 12 – 18 🌟
We are back with another round of appreciation for members who lit up the Databricks Community this week with their contributions, positivity, and support! Here are the Sparks for this week: nayan_wylde – Took initiative in answering questions with...
- 44 Views
- 0 replies
- 1 kudos
- 229 Views
- 1 replies
- 1 kudos
Issue with Lakebridge transpile installation – SSL Certificate Error
Hi Team,I am trying to use Lakebridge to test a small piece of code for conversion. The base installation of Lakebridge worked fine, but when I attempt to install transpile, I encounter SSL-related errors. I even tried to hardcode of the certificates...
- 229 Views
- 1 replies
- 1 kudos
Latest Reply
- 1 kudos
The error indicates that while installing Lakebridge's transpile component on Databricks with Python 3.13, SSL certificate verification fails due to a "Missing Authority Key Identifier" in the certificate chain. This is a result of stricter requireme...
- 1 kudos
- 91 Views
- 3 replies
- 0 kudos
Inconsistent behavior of LakeBridge transpiler for similar scripts
Hi Everyone,I am testing the LakeBridge prototype and noticed inconsistent behavior when converting stored procedures.For simple scripts, the conversion is correct.But for medium/complex scripts, especially those with multiple LEFT JOINs and column e...
- 91 Views
- 3 replies
- 0 kudos
Latest Reply
- 0 kudos
Is there a way to standardize/optimize the conversion so it doesn’t blow up into thousands of lines? Yes, there are actionable methods to standardize and optimize LakeBridge conversions to prevent code from ballooning into thousands of lines, but t...
- 0 kudos
- 106 Views
- 1 replies
- 2 kudos
Vacuuming clones in USER_ISOLATION mode and ThreadPool Executor
Hello,I run my process on a shared, interactive cluster (data security mode: USER_ISOLATION).I run operations on multiple tables having each of them as a separate thread, pseudo-code :try: truncate target_tablevacuum target_table (retain 0 hours with...
- 106 Views
- 1 replies
- 2 kudos
Latest Reply
- 2 kudos
This is most likely a bug. It certainly is unexpected and should be reported to Databricks support or your platform administrator for clarification and remediation. As a temporary workaround, running vacuum commands outside of threaded contexts or s...
- 2 kudos
- 14 Views
- 0 replies
- 0 kudos
Establish connectivity from databricks serverless to Cloud SQL (GCP) Postgres database
Hi team,We have setup workspace using databricks managed network (databricks is setup on GCP). There is a cloud sql db in customer managed network. We want to establish connectivity from databricks serverless to database. Could someone please tell di...
- 14 Views
- 0 replies
- 0 kudos
- 2433 Views
- 8 replies
- 2 kudos
Resolved! Reporting serverless costs to azure costs
So, we've just recently applied serverless budget polices to some of our vector searches and apps. At the moment they're all going to azure under one general tag that we created.However, we needed more definition. So i added the serverless budget pol...
- 2433 Views
- 8 replies
- 2 kudos
Latest Reply
- 2 kudos
Billing or set up explicit export pipelines. Check whether your serverless budget policy tags are under a different namespace in Azure, as sometimes they show up nested.
- 2 kudos
- 3455 Views
- 5 replies
- 0 kudos
Delta Live tables stream output to Kafka
Hello,Wanted to Know if we can write the stream output to a Kafka topic in DLT pipeline?Please let me know.Thankyou.
- 3455 Views
- 5 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi ! Ensure your code is set up to use these libraries. Here is the complete example: Navigate to your cluster configuration:Go to your Databricks workspace.Click on "Clusters" and select your cluster.Go to the "Libraries" tab. Install the necessar...
- 0 kudos
Announcements
| User | Count |
|---|---|
| 2088 | |
| 1056 | |
| 941 | |
| 731 | |
| 636 |
Latest from our Blog
Normalised ingestion of Vehicle Telematics
Introduction Internet of Things (IoT) technology has increased real-time transmission, processing, and analysis of sensor data. In the automotive industry, this manifests as vehicles equipped with sen...
Performance Showdown: withColumn vs withColumns in Apache Spark
Introduction When building scalable data pipelines in Apache Spark, the way you add or transform columns in a DataFrame can have a dramatic impact on performance. In this post, we’ll dive deep into th...
[CUSTOMER BLOG] Scaling Geospatial Analytics: Efficient Visualization of Massive Spatial Datasets
Introduction If you’ve ever tried visualizing vast geospatial datasets—such as billions of points—you’ve likely run into performance bottlenecks or high costs. At S&P Global Commodity Insights, we set...
FinOps and Databricks System Tables
Original post published on 30 May 2024, here.FinOps and Databricks System Tables Dear all, I have to admit that FinOps and System Tables are not the most interesting topic to talk about it, but everyt...
