Learn, Level Up, and Get Rewarded! Save the dates: October 10 – October 31, 2025! We're celebrating learning, growth, and career advancement with an exciting opportunity to level up in data engineering, data analysis, machine learning, and genera...
- 18460 Views
- 20 replies
- 18 kudos
4 weeks ago
We are excited to kick off our Weekly Active Members series — a way to recognize and appreciate those who make our Community thrive.This recognition focuses not on the number of posts, but on accepted solutions and quality contributions that genuinel...
- 291 Views
- 6 replies
- 11 kudos
Friday
July 2025 Release Highlights ️Data Engineering Moving tables between the Lakeflow declarative pipeline is GA Teams can seamlessly reorganize streaming tables and materialized views as needs change—split a pipeline, consolidate workloads,...
- 58 Views
- 0 replies
- 4 kudos
yesterday
Embark on a hands-on learning journey that will help deliver the data and AI-driven outcomes that matter most to your business. Experience the Databricks EMEA Learning Festival to explore proven strategies and practical steps that will speed up t...
- 220 Views
- 1 replies
- 2 kudos
Friday
Why Lakebase + Databricks Apps Lakebase and Databricks Apps work together to simplify full-stack development on the Databricks platform: Lakebase gives you a fully managed Postgres database with fast reads, writes, and updates, plus modern features l...
- 144 Views
- 0 replies
- 3 kudos
Friday
Community Activity
- 521 Views
- 8 replies
- 3 kudos
Maven libraries in VNet injected, UC enabled workspace on Standard Access Mode Cluster
Hi!As the title suggests, I want to install Maven libaries on my cluster with access mode 'Standard'. Our workspace is VNet injected and has Unity Catalog enabled.The coordinates have been allowlisted by the account team according to these instructio...
- 521 Views
- 8 replies
- 3 kudos
Latest Reply
- 3 kudos
@nayan_wylde @szymon_dybczak I just tried using a JAR I uploaded to an allowlisted Volume (ojdbc8 of oracle) and I get the same error. it seems like I'm able to install an JAR, but once it's installed my cluster is broken.
- 3 kudos
- 33669 Views
- 62 replies
- 2 kudos
Resolved! Databricks exam got suspended. Need urgent help
Hello Team, I faced multiple interruptions and challenges while attempting my certification, The exam got paused 4 times where twice I had to show my desk and my entire room. Eventually my exam got suspended. I was not involved in any unfair means an...
- 33669 Views
- 62 replies
- 2 kudos
Latest Reply
- 2 kudos
@cert-ops Today my Databricks exam got suspended without reason. I set my system to maximum brightness, and initially the proctor accepted it. But after some time, without any changes in lighting or brightness, it was suddenly suspended.Need your hel...
- 2 kudos
- 68 Views
- 1 replies
- 0 kudos
Passing a list of images on AI_Query
HiI would like to know how can I pass a list of images to ai_query on the same call, It will be helpful to extract some metadata information on a multipage document, where I want to know if different images to see if it belongs to the same document.T...
- 68 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Hello @Andreyai Has your issue got resolved?
- 0 kudos
- 47 Views
- 4 replies
- 0 kudos
transformWithStateInPandas throws "Spark connect directory is not ready" error
Hello,we employ arbitrary stateful aggregations in our data processing streams on Azure Databricks, and would like to migrate from applyInPandasWithState to transformWithStateInPandas. We employ the Python API throughout our solution, and some of our...
- 47 Views
- 4 replies
- 0 kudos
Latest Reply
- 0 kudos
https://www.databricks.com/blog/introducing-transformwithstate-apache-sparktm-structured-streamingHere they specifically mention Unity Catalog clusters (see Availability section), even though in the release notes this is not mentioned as a requiremen...
- 0 kudos
- 22 Views
- 0 replies
- 0 kudos
Duplicated storage location
Hello there!After a year working with Databricks, we have found that we could improve our catalog and schemas organization, so we would like to perform a 'migration' process where we move tables, volumes, models, etc from one catalog to another.The p...
- 22 Views
- 0 replies
- 0 kudos
- 28 Views
- 1 replies
- 0 kudos
Ingest data from SQL Server
I've been working on data ingestion from SQL Server to UC using lakeflow connect. Lakeflow connect actually made the work easier when everything is right. I am trying to incorporate this with DAB and this would work fine with schema and table tags fo...
- 28 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Hello @Vamsi_S Good day! Did you Preprocessing Table Names in CI/CD and Generate YAML Dynamically (Recommended for Dynamic, Automated Ingestion)Did you contact your databricks account manager (incase if you working with a company) for feature request...
- 0 kudos
- 34 Views
- 2 replies
- 2 kudos
How to load all the previous day's data only into the newly added column of the existing delta table
How to load all the previous day's data only into the newly added column of the existing delta table? Is there any option available to do that without writing any logic?
- 34 Views
- 2 replies
- 2 kudos
Latest Reply
- 2 kudos
@shan-databricks there's certainly ways for the schema to evolve within your delta tables that's supported out of the box: https://docs.databricks.com/aws/en/delta/update-schema#enable-schema-evolution To update older records, they'd likely have NULL...
- 2 kudos
- 26 Views
- 2 replies
- 1 kudos
cosmosdb metadata integration with unity catalog
Hi Team,How can we integrate Cosmos DB Meta data with Unity Catalog, can you please provide some insights on this?Regards,Phani
- 26 Views
- 2 replies
- 1 kudos
Latest Reply
- 1 kudos
Hello @Phani1 Good day:I have found a whole document on your requirementshttps://community.databricks.com/t5/technical-blog/optimising-data-integration-and-serving-patterns-with-cosmos-db/ba-p/91977 It has a project with it as well.
- 1 kudos
- 37 Views
- 3 replies
- 13 kudos
Level Up Your Databricks Game - Episode 1: Widgets
Hi all,TheOC here with my first of (hopefully!) many blogs on the Databricks Community. I'm hoping in this series to share quick, practical tips to help you get the most out of Databricks. Today's topic is: Widgets.If you're anything like me, you've ...
- 37 Views
- 3 replies
- 13 kudos
Latest Reply
- 13 kudos
I haven't used widgets much before, but I also hadn't realised the power of them. I will certainly be taking advantage from now on. I am curious if there is a specific date widget that allows you to select a date from a calendar-like table? If not, t...
- 13 kudos
- 32 Views
- 1 replies
- 0 kudos
Resolved! How to build Data Pipeline to consume data from Adobe Campaign to Azure Databricks
May Techie please help me design the pipeline with Databricks.I don't have any control over Adobe.How to set up a data pipeline that moves csv files from Adobe to ADLS Gen2 via a cron job, using Databricks.where this cron job will execute ? how ADLS ...
- 32 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Hello @Datalight Good day!Can I please know what do you mean by "you dont have any control over adobe"? I found similar case study over here: https://learn.microsoft.com/en-us/answers/questions/5533633/data-pipeline-to-push-files-from-external-system...
- 0 kudos
- 484 Views
- 8 replies
- 3 kudos
Cannot deploy DAB with the Job branch using a feature branch in Workspace UI
Hi, I tried to deploy DAB on Workspace UI with a feature branch (sf-trans-seq) targeted to Dev. After deploying successfully, the Job branch is, however, using the master branch (see the screenshot below).Is there any option to force the Job branch t...
- 484 Views
- 8 replies
- 3 kudos
Latest Reply
- 3 kudos
I agree.Can you mark your (or someone else´s) answer as solved? Because I think you won´t be the only one with this issue/feature.
- 3 kudos
- 58 Views
- 0 replies
- 4 kudos
What's new in Databricks: July - August 2025
July 2025 Release Highlights ️Data Engineering Moving tables between the Lakeflow declarative pipeline is GA Teams can seamlessly reorganize streaming tables and materialized views as needs change—split a pipeline, consolidate workloads,...
- 58 Views
- 0 replies
- 4 kudos
- 60 Views
- 3 replies
- 0 kudos
Databricks Data Engineer Associate Exam Got suspended
Hi team,My Databricks Certified Data Engineer Associate exam got suspended before 10 minutes left to submission. I have answered all the questions and was reviewing them. My exam got suspended due to eye movement. I was not looking away from laptop s...
- 60 Views
- 3 replies
- 0 kudos
Latest Reply
- 0 kudos
Hello @jaspreetkaur,Thank you for filing a ticket with our support team, Support team will respond shortly. Please note that we cannot provide support or handle exam suspensions via community.Thanks & Regards,@cert-ops
- 0 kudos
- 177 Views
- 8 replies
- 0 kudos
Exam got suspended on 30-Aug 2025 , Please reschedule it
Hello Team, I faced multiple interruptions and challenges while attempting my certification, The exam got paused 1 time and I waited for half an hour for someone to reach out to me from the support team. Eventually my exam got suspended without any w...
- 177 Views
- 8 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi @Cert-Team @Cert-TeamOPS My exam got suspended of the exam. I have worked really hard for this exam. I have followed the instructions clearly but still my exam got suspended, I am not sure why. Please let me take my exam. Please help me to resche...
- 0 kudos
- 291 Views
- 6 replies
- 37 kudos
🌟 Community Spark of the Week | Aug 22 – Aug 28 🌟
We are excited to kick off our Weekly Active Members series — a way to recognize and appreciate those who make our Community thrive.This recognition focuses not on the number of posts, but on accepted solutions and quality contributions that genuinel...
- 291 Views
- 6 replies
- 37 kudos
Latest Reply
- 37 kudos
Congratulations to the Sparks of the week!Your contributions keep those transistors tick
- 37 kudos
Announcements
Latest from our Blog
Auto Loader: Data Ingestion with built-in Data Audit & Validation
Introduction: Auto Loader as the Go-To Ingestion Solution Databricks Auto Loader has become the standard for incremental data ingestion in cloud environments. It provides automatic schema inference, h...
Apache Spark 4.0 – A New Era for Scalable Machine Learning and AI
As a machine learning practitioner, I’m incredibly excited about the release of Apache Spark 4.0 (officially announced May 23, 2025). This landmark update brings a bevy of new features and improvement...
NVIDIA’s AI-Native Digital Twin on Databricks, True AI Democratization for Telecom I
The telecommunications industry is on the verge of transformation, driven by the emergence of AI-native digital twins and open-source simulation tools that Nvidia released during the GTC 2025 conferen...
My First Experience at the Data + AI Summit: Highlights and Reflections
Attending the Data + AI Summit 2025 in San Francisco in person for the first time was an exceptional experience. As a Databricks MVP, author, instructor, and data engineer with over a decade of experi...
