cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

yasmeen
by • Contributor
  • 90 Views
  • 1 replies
  • 5 kudos

What's Newly GA in Databricks: June to November 2026

Databricks shipped a lot between June and November 2026, and it's easy to lose track of what is generally available versus still in preview. Just went through the official release notes for that period and pulled out the GA features that matter most ...

  • 90 Views
  • 1 replies
  • 5 kudos
Latest Reply
amee094huyen
  • 5 kudos

Hello, It's really helpful update

  • 5 kudos
alejandrorr
by • Visitor
  • 100 Views
  • 3 replies
  • 0 kudos

Retries not triggered for a Run Job task

Is it possible to configure retries for a Databricks task of type Run Job?I configured max_retries and min_retry_interval_millis on the task in a Databricks Asset Bundle, as I would for a notebook task. However, when the called job fails, the Run Job...

  • 100 Views
  • 3 replies
  • 0 kudos
Latest Reply
ashishshivatare
Databricks Partner
  • 0 kudos

Hi alejandrorrThanks for checking all of that, it rules out the bundle. bundle validate shows the retry fields at the task level, the deployed bundle files have them, and the Jobs API returns the task without them. So the backend is silently dropping...

  • 0 kudos
2 More Replies
Mateus-Moreira
by • New Contributor II
  • 130 Views
  • 2 replies
  • 2 kudos

Databricks Cost Monitoring: Sharing My SQL Query – Feedback and Suggestions Welcome!

Hi everyone!I've been working on a SQL query to monitor and analyze Databricks usage costs using Unity Catalog system tables.The goal is to consolidate usage information from different workspaces and identify costs by resource, such as Jobs, Apps, SQ...

  • 130 Views
  • 2 replies
  • 2 kudos
Latest Reply
amalik23
New Contributor II
  • 2 kudos

Nice starting point. One thing I’d change is not filtering out short-lived job clusters.Those clusters may be temporary, but the spend is real. I’d keep all billing records and map them back to something more stable: job, pipeline, application, team,...

  • 2 kudos
1 More Replies
dgr
by • New Contributor II
  • 148 Views
  • 3 replies
  • 0 kudos

Guidance on resolving Delta Lake concurrency exceptions during parallel writes

Hi everyone,We are new to Databricks and are currently in the process of migrating our ETL workloads from SSIS (SQL Server) to Databricks Delta tables.Coming from a traditional relational database background where row-level locking handles parallel u...

  • 148 Views
  • 3 replies
  • 0 kudos
Latest Reply
amalik23
New Contributor II
  • 0 kudos

I’d still keep a small retry policy, but I wouldn’t rely on retries as the concurrency strategy.If row-level concurrency is enabled and the jobs truly update different rows, those writes should normally be able to proceed independently. Retries are s...

  • 0 kudos
2 More Replies
MichaelAnckaert
by • New Contributor
  • 288 Views
  • 6 replies
  • 6 kudos

Issue with CDC ingestion pipeline

Hello everyoneI'm trying to setup an ingestion pipeline from a mysql server and keep running into an issue. The setup was validated and the pipeline created. When I run the pipeline the destination tables are created in the schema, but then the follo...

  • 288 Views
  • 6 replies
  • 6 kudos
Latest Reply
MichaelAnckaert
New Contributor
  • 6 kudos

I managed to get the CDC pipeline working! The staging catalog had an S3 storage attached, the schema did not but according to the UI it inherited from the catalog. I then added a storage root to the metastore and, after recreating the staging catalo...

  • 6 kudos
5 More Replies
Albertino
by • New Contributor II
  • 844 Views
  • 4 replies
  • 2 kudos

databricks-connect library for python and pandas 3

Hello,databricks-connect is pinning pandas during the installation. Since we're moving towards pandas 3 can you please add the support for the newest version as well?

Data Engineering
databricks-connect
Pandas
python
  • 844 Views
  • 4 replies
  • 2 kudos
Latest Reply
SergejK
New Contributor
  • 2 kudos

Hi, it seems pyspark already has support for pandas 3.0 to some extend. So this might be solvable now. I am already using 3.x.

  • 2 kudos
3 More Replies
khacdat-dang
by • New Contributor
  • 165 Views
  • 2 replies
  • 1 kudos

Resolved! Git folder fails with "Input/output error" on .git/index and edited files, even after a fresh clone

Hi all,I'm hitting persistent Git errors in Databricks Git folders (Repos) and would appreciate any advice.EnvironmentCloud: AWSCompute: serverless onlyRepo: private GitHub repository, working on a develop branchContents: a Declarative Automation Bun...

  • 165 Views
  • 2 replies
  • 1 kudos
Latest Reply
khacdat-dang
New Contributor
  • 1 kudos

Thank you so much !It finally worked for me !!

  • 1 kudos
1 More Replies
tabinashabir
by • New Contributor II
  • 108 Views
  • 1 replies
  • 0 kudos

Zerobus Ingest availability in Azure South Africa North (southafricanorth)?

We are looking at Zerobus Ingest for streaming events directly into Unity Catalog Delta tables, where the delta table is the only consumer. Our Azure Databricks workspace is in South Africa North (southafricanorth), and data residency means we cannot...

  • 108 Views
  • 1 replies
  • 0 kudos
Latest Reply
arthurfr23
New Contributor III
  • 0 kudos

You're right, southafricanorth isn't currently on the Zerobus Ingest regional availability list for Azure. The Zerobus endpoint always runs in your workspace's region, so you can't point clients at an endpoint in another region and still write to you...

  • 0 kudos
inaafina
by • New Contributor
  • 144 Views
  • 2 replies
  • 0 kudos

Overwrite Job Errors

I keep on getting this error message when trying to create a parquet in my notebook. Has anyone found this error too? My overwrite path was working perfectly fine before.SparkConnectGrpcException: (java.lang.IllegalStateException) operationId: d5871d...

  • 144 Views
  • 2 replies
  • 0 kudos
Latest Reply
saurabh18cs
Honored Contributor III
  • 0 kudos

Hi  @inaafina setup like this , this is a new way to automatically resolve right package versions based on your compute. you have to use databricks cli  e.g. databricks environments setup-local \ --serverless-version 5 \ --dry-run

  • 0 kudos
1 More Replies
DB1To3
by • Valued Contributor II
  • 559 Views
  • 10 replies
  • 6 kudos

Resolved! Missing spark structured streaming metadata blobs?

I'm exploring spark structured streaming, and contemplating uses for it.I'd like to launch a job on a recurring basis from an external scheduler.  It will launch a spark job with trigger (availableNow=True). I'm concerned about overlapping executions...

DB1To3_0-1794341533309.png
  • 559 Views
  • 10 replies
  • 6 kudos
Latest Reply
saurabh18cs
Honored Contributor III
  • 6 kudos

Hi @DB1To3 Just make this concurrent run to 1 for your streaming jobs are your problem is sorted , it wont corrupt checkpoint locatiion  

  • 6 kudos
9 More Replies
DB1To3
by • Valued Contributor II
  • 207 Views
  • 4 replies
  • 1 kudos

Resolved! Statestore is not valid when using Spark Structured Streaming

I'm using spark structured streaming and it works properly at first.  But when I try to audit the checkpointLocation I can't get it to work.  It seems like it should be a relatively simple task to query the statestore and audit the processing of a st...

DB1To3_0-1794251574660.png
  • 207 Views
  • 4 replies
  • 1 kudos
Latest Reply
SajeevPS
New Contributor II
  • 1 kudos

I think the error is expected for this pipeline. The state reader functions (read_statestore / read_state_metadata) are for stateful Structured Streaming queries, meaning ones with a stateful operator such as aggregations, dropDuplicates, stream-stre...

  • 1 kudos
3 More Replies
rajkumar8k
by • New Contributor III
  • 198 Views
  • 2 replies
  • 6 kudos

Databricks Latest Features: Genie Code CLI, App Builder, Agent Memory and DABs Deployment History

Hi everyone,Databricks has shipped a lot of new features recently. Here’s a quick summary of the most useful ones, grouped by area.1. DABs Deployment HistoryWhat it is:Databricks Asset Bundles (DABs) now keep a history of every deployment. Each recor...

  • 198 Views
  • 2 replies
  • 6 kudos
Latest Reply
Jaswanth_alla
New Contributor
  • 6 kudos

Great summary! The features that stand out most to me are DABs Deployment History and UC JDBC Connections. Both solve real operational challenges around governance, visibility, and maintainability. Exciting to see Databricks continuing to simplify pl...

  • 6 kudos
1 More Replies
AlexMorgan
by • New Contributor
  • 321 Views
  • 4 replies
  • 1 kudos

How do you test data quality in a Delta Lake pipeline?

I'm exploring practical approaches to validating data quality in Databricks pipelines.For pipelines that ingest data into Delta tables, what checks do you typically automate at each stage?I'm particularly interested in handling:Schema changes and sch...

  • 321 Views
  • 4 replies
  • 1 kudos
Latest Reply
rajkumar8k
New Contributor III
  • 1 kudos

In my experience a combination works best:DLT / Lakeflow expectations check data quality while the pipeline runs. Use warn, drop or fail depending on how serious the issue is, and send bad records to a quarantine table instead of losing them.PySpark ...

  • 1 kudos
3 More Replies
2nonto
by • New Contributor
  • 162 Views
  • 1 replies
  • 0 kudos

Pushing a notebook with outputs to Github

I am experiencing an error when trying to commit output and Databricks notebook to GitHub. The error message states, "Error creating or updating commit outputs config file."Could you please assist me in resolving this issue?

Screenshot 2026-11-11 214121.png
  • 162 Views
  • 1 replies
  • 0 kudos
Latest Reply
Yougasundar
Contributor
  • 0 kudos

Hi @2nonto,That message comes from the step that creates the .databricks/commit_outputs file, which controls whether the outputs of .ipynb notebooks are committed. Please check these points in order. 1) Committing outputs is off by default. A workspa...

  • 0 kudos
Labels