Celebrate Data and AI Summit with us at the Virtual Learning Festival! Mark your calendars: June 11 - July 2, 2025! In honor of our Annual Data and AI Summit, we’re offering an exciting opportunity to upskill in data engineering, data analysis, m...
- 9419 Views
- 8 replies
- 4 kudos
a week ago
Be part of a global movement! Connect with 22,000 data enthusiasts at 700+ sessions, keynotes, and hands-on training at this year’s Data + AI Summit. Whether you’re into data intelligence, governance, AI, or data warehousing, this is your chance to l...
- 14296 Views
- 4 replies
- 6 kudos
02-19-2025
Flash Sale — use promo code: TRNGTM0Q to get 50% off training. Courses are now just $212.50 each. Act now – the offer is valid through May 23. This year, Databricks Training and Certification returns to the Data + AI Summit in San Francisco from June...
- 331 Views
- 0 replies
- 0 kudos
Monday
Today we’re announcing the launch of Data Intelligence for Marketing, combining the Databricks Data Intelligence Platform with out-of-the-box integrations to an ecosystem of leading marketing solution providers. Here, teams can bring customer and cam...
- 611 Views
- 0 replies
- 0 kudos
a week ago
Liquid Clustering is an innovative data management technique that significantly simplifies your data layout-related decisions. You only have to choose clustering keys based on query access patterns. Thousands of customers have benefited from better q...
- 758 Views
- 0 replies
- 0 kudos
a week ago
Over the past several months, we’ve made DLT pipelines faster, more intelligent, and easier to manage at scale. DLT now delivers a streamlined, high-performance foundation for building and operating reliable data pipelines at any scale. First, we’re ...
- 773 Views
- 0 replies
- 0 kudos
a week ago
Community Activity
- 41 Views
- 4 replies
- 0 kudos
Not able create and start a cluster
Hi Team,I am trying to use community edition for learning below is my URL detailshttps://community.cloud.databricks.com/compute/interactive?o=2059657917292434Due to some reason my clusters are taking nearly 45 to 60 minutes for creation and after if ...
- 41 Views
- 4 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi @nikhilj0421 still the same issue, i could see its working till spark 15.4 LTS above 15.4LTS whatever version i useits not woring i am not able to create any cluster.
- 0 kudos
- 0 Views
- 0 replies
- 0 kudos
Celebrate the Data + AI Summit with Us! | Virtual Learning Festival: June 11 - July 2, 2025
Hey Austin Databricks community! To celebrate the Annual Data + AI Summit, Databricks is rolling out an amazing Virtual Learning Festival packed with self-paced learning pathways across:Data EngineeringData AnalysisMachine LearningGenerative AIWhat’s...
- 0 Views
- 0 replies
- 0 kudos
- 176 Views
- 1 replies
- 0 kudos
Spark Streaming Job gets stuck in the "Stream Initializing"
Hello all,I am having an issue with my Spark Streaming Job. It is stuck at "Stream Initializing" stage.Need your help here to understand what is happening inside the "Stream Initializing" stage of Spark Streaming job which is taking so long. Here are...
- 176 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi @hao-uit, do you see any kind of load on the driver and event logs? Also, what libraries you have installed on your cluster?
- 0 kudos
- 2 Views
- 0 replies
- 0 kudos
Delta Live Table (Streaming Tables) for excel (.xlsx, .xls)
What's the native way to ingest excel files using a streaming table? I wish that when the excel files land in unity catalog, it can pick up those and load it in to the Streaming Table. Data is Small, so we can afford some kind of UDF, but we really n...
- 2 Views
- 0 replies
- 0 kudos
- 60 Views
- 1 replies
- 0 kudos
Anyone using Databricks Query Federation for ETL purpose ?
Hello All,We have a use case to fetch data from a SQL Server wherein we have some tables to consume. This is typically a OLTP setup wherein the comes in a regular interval. Now, as we have Unity Catalog enabled, we are interested in exploring Databr...
- 60 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi @dipanjannet, you can leverage DLT feature to do so. Please check: https://docs.databricks.com/aws/en/dlt/transform https://docs.databricks.com/aws/en/dlt/stateful-processing Here is the step-by-step tutorial: https://docs.databricks.com/aws/en/d...
- 0 kudos
- 44646 Views
- 7 replies
- 6 kudos
how to read the CSV file from users workspace
Hi!I have been carrying out a POC, so I created the CSV file in my workspace and tried to read the content using the techniques below in a Python notebook, but did not work.Option1:repo_file = "/Workspace/Users/u1@org.com/csv files/f1.csv"tmp_file_na...
- 44646 Views
- 7 replies
- 6 kudos
Latest Reply
- 6 kudos
Hi @Dev ,Generally, What happens spark reader APIs point to the DBFS by default. And, to read the file from User workspace, we need to append 'file:/' in the prefix.Thanks
- 6 kudos
- 8 Views
- 0 replies
- 0 kudos
issue on databricks bundle deploy
HiI am trying to configure Databricks Asset Bundle, but got error on deploymentDatabricks bundle init ----------- OKDatabricks bundle validate ----- OKDatabricks bundle deploy ------ Failerror : PS C:\Databricks_DABs\DABs_Init\DABS_Init> databricks b...
- 8 Views
- 0 replies
- 0 kudos
- 314 Views
- 6 replies
- 0 kudos
Multiple concurrent jobs using interactive cluster
Hi All,I have notebook in Databricks. This notebook is executed from azure datafactory pipeline having a databricks notebook activity with linkedservice connected to an interactive cluster.When multiple concurrent runs of this pipeline are created, I...
- 314 Views
- 6 replies
- 0 kudos
- 5 Views
- 0 replies
- 0 kudos
Universal/cross-tab filters in dashboard
With the recent addition of dashboard tabs in databricks, I could not find a way to have filters apply to multiple tabs within the same dashboard - so far I have had to manually create and apply filters to each of my tabs individually.
- 5 Views
- 0 replies
- 0 kudos
- 6 Views
- 0 replies
- 0 kudos
An Equivalent Implementation of XIRR in Excel within PySpark
IntroductionIRR is a common method used by financial personnel to evaluate the financial benefits of project investment, and its formula is as follows.In excel, you can use the XIRR function to calculate the IRR of non-periodic cash flows easily.But ...
- 6 Views
- 0 replies
- 0 kudos
- 49 Views
- 1 replies
- 0 kudos
Databricks JDBC Driver Version 42 Limitations
We found that the Databricks JDBC driver does not support:Connection.setAutoCommit(false)Connection.commit()Connection.rollback()Execution of BEGIN TRANSACTIONCan you help us understand why these operations are not supported by the Databricks JDBC dr...
- 49 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi @Ankit_Kothiya As far as I know, the Databricks JDBC driver doesn’t support manual transaction controls like setAutoCommit(false), commit(), rollback(), or BEGIN TRANSACTION. That’s because Delta Lake are designed to support atomic operations at t...
- 0 kudos
- 2172 Views
- 2 replies
- 0 kudos
Workflow UI not showing all jobs that were run
I found a scenario where the workflow UI seems to not list all the jobs that ran.The objective is to keep trying to run a job, using the job rest api, when the service returns a status of "skipped" while another run is currently running, until the jo...
- 2172 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
I'm.facing the same issue I used to see the jobs ran via databricks bundle but now I can't see those jobs ? Can ay please specify wat went wrong ?
- 0 kudos
- 33 Views
- 1 replies
- 0 kudos
Notebook ID in DESCRIBE HISTORY not showing
We've recently installed Databricks 14.3 LTS with Unity Catalog and for some reason that is escaping me the Notebook Id is not showing up when I execute the DESCRIBE HISTORY SQL command. Example below for table test_catalog.lineagedemo.lm_lineage_tes...
- 33 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi @lumen, Generally, notebook Id will be populated if the version was created via notebook. I suspect these versions are created by other means? Here is an example where I created first 2 versions via a notebook and the 3rd version created via SQL E...
- 0 kudos
- 15 Views
- 0 replies
- 0 kudos
Email notification to end users
Is there a way a way we can notify all of our databricks end users by email when there is a issue? We currently have our jobs setup to notify the technical team when a job workflow fails. That part works fine.But we would like the ability to maybe us...
- 15 Views
- 0 replies
- 0 kudos
- 677 Views
- 7 replies
- 1 kudos
parameterized ALTER TABLE SET TAGS
I would like to use parametrized sql queries to run SET TAGs commands on tables, but cannot figure out how to parameterize the query to prevent SQL injection. Both the `?` and `:key` parameter syntaxes throw a syntax errorBasically, I'd like to do th...
- 677 Views
- 7 replies
- 1 kudos
Announcements
Latest from our Blog
Unlocking the future of the Automotive Industry (Part 1) - Scalable Geospatial Analytics
Introduction The automotive industry is undergoing a data-driven transformation, using vast volumes of geospatial, telematics, and sensor data to power AI innovation. As vehicles, infrastructure, and ...
How to integrate Mistral AI multimodal models from Huggingface with Databricks
Introduction Multimodal AI models can process multiple types of data simultaneously, such as images, audio and text. This capability enables more intuitive interactions with AI systems like chatbots a...
How to perform CDC Data Load and SCD Type II with Databricks Lakeflow Connect and Pipelines
Introduction Every data warehousing project once defined and deemed necessary contains three primary tasks: Ingest data from source.Populate and transform data for reportingLeverage Slowly Changing ...
Managing Machine Learning Library Dependency on Databricks
1. The Role of Dependencies in MLOps2. Selecting the Optimal Cluster Type3. Shift Toward Integrated Platform-Native MLOps4. Platform-Native Approaches4.1. Cluster-Scoped Library Management4.2. Efficie...
