cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

2nonto
by • New Contributor
  • 168 Views
  • 2 replies
  • 1 kudos

Pushing a notebook with outputs to Github

I am experiencing an error when trying to commit output and Databricks notebook to GitHub. The error message states, "Error creating or updating commit outputs config file."Could you please assist me in resolving this issue?

Screenshot 2026-11-11 214121.png
  • 168 Views
  • 2 replies
  • 1 kudos
Latest Reply
saurabh18cs
Honored Contributor III
  • 1 kudos

Hi @2nonto then also good to check following :  

  • 1 kudos
1 More Replies
MichaelAnckaert
by • New Contributor
  • 310 Views
  • 7 replies
  • 6 kudos

Issue with CDC ingestion pipeline

Hello everyoneI'm trying to setup an ingestion pipeline from a mysql server and keep running into an issue. The setup was validated and the pipeline created. When I run the pipeline the destination tables are created in the schema, but then the follo...

  • 310 Views
  • 7 replies
  • 6 kudos
Latest Reply
suranga
Databricks Partner
  • 6 kudos

Great to hear it's running, and thanks for coming back with what fixed it. Adding the metastore storage root and then recreating the staging catalog is a useful detail for the next person who hits "s3:/".If you get a minute, marking the reply that he...

  • 6 kudos
6 More Replies
yasmeen
by • Contributor
  • 109 Views
  • 1 replies
  • 5 kudos

What's Newly GA in Databricks: June to November 2026

Databricks shipped a lot between June and November 2026, and it's easy to lose track of what is generally available versus still in preview. Just went through the official release notes for that period and pulled out the GA features that matter most ...

  • 109 Views
  • 1 replies
  • 5 kudos
Latest Reply
amee094huyen
  • 5 kudos

Hello, It's really helpful update

  • 5 kudos
alejandrorr
by • Visitor
  • 110 Views
  • 3 replies
  • 0 kudos

Retries not triggered for a Run Job task

Is it possible to configure retries for a Databricks task of type Run Job?I configured max_retries and min_retry_interval_millis on the task in a Databricks Asset Bundle, as I would for a notebook task. However, when the called job fails, the Run Job...

  • 110 Views
  • 3 replies
  • 0 kudos
Latest Reply
ashishshivatare
Databricks Partner
  • 0 kudos

Hi alejandrorrThanks for checking all of that, it rules out the bundle. bundle validate shows the retry fields at the task level, the deployed bundle files have them, and the Jobs API returns the task without them. So the backend is silently dropping...

  • 0 kudos
2 More Replies
Mateus-Moreira
by • New Contributor II
  • 139 Views
  • 2 replies
  • 2 kudos

Databricks Cost Monitoring: Sharing My SQL Query – Feedback and Suggestions Welcome!

Hi everyone!I've been working on a SQL query to monitor and analyze Databricks usage costs using Unity Catalog system tables.The goal is to consolidate usage information from different workspaces and identify costs by resource, such as Jobs, Apps, SQ...

  • 139 Views
  • 2 replies
  • 2 kudos
Latest Reply
amalik23
New Contributor II
  • 2 kudos

Nice starting point. One thing I’d change is not filtering out short-lived job clusters.Those clusters may be temporary, but the spend is real. I’d keep all billing records and map them back to something more stable: job, pipeline, application, team,...

  • 2 kudos
1 More Replies
dgr
by • New Contributor II
  • 150 Views
  • 3 replies
  • 0 kudos

Guidance on resolving Delta Lake concurrency exceptions during parallel writes

Hi everyone,We are new to Databricks and are currently in the process of migrating our ETL workloads from SSIS (SQL Server) to Databricks Delta tables.Coming from a traditional relational database background where row-level locking handles parallel u...

  • 150 Views
  • 3 replies
  • 0 kudos
Latest Reply
amalik23
New Contributor II
  • 0 kudos

I’d still keep a small retry policy, but I wouldn’t rely on retries as the concurrency strategy.If row-level concurrency is enabled and the jobs truly update different rows, those writes should normally be able to proceed independently. Retries are s...

  • 0 kudos
2 More Replies
Albertino
by • New Contributor II
  • 855 Views
  • 4 replies
  • 2 kudos

databricks-connect library for python and pandas 3

Hello,databricks-connect is pinning pandas during the installation. Since we're moving towards pandas 3 can you please add the support for the newest version as well?

Data Engineering
databricks-connect
Pandas
python
  • 855 Views
  • 4 replies
  • 2 kudos
Latest Reply
SergejK
New Contributor
  • 2 kudos

Hi, it seems pyspark already has support for pandas 3.0 to some extend. So this might be solvable now. I am already using 3.x.

  • 2 kudos
3 More Replies
khacdat-dang
by • New Contributor
  • 169 Views
  • 2 replies
  • 1 kudos

Resolved! Git folder fails with "Input/output error" on .git/index and edited files, even after a fresh clone

Hi all,I'm hitting persistent Git errors in Databricks Git folders (Repos) and would appreciate any advice.EnvironmentCloud: AWSCompute: serverless onlyRepo: private GitHub repository, working on a develop branchContents: a Declarative Automation Bun...

  • 169 Views
  • 2 replies
  • 1 kudos
Latest Reply
khacdat-dang
New Contributor
  • 1 kudos

Thank you so much !It finally worked for me !!

  • 1 kudos
1 More Replies
tabinashabir
by • New Contributor II
  • 112 Views
  • 1 replies
  • 0 kudos

Zerobus Ingest availability in Azure South Africa North (southafricanorth)?

We are looking at Zerobus Ingest for streaming events directly into Unity Catalog Delta tables, where the delta table is the only consumer. Our Azure Databricks workspace is in South Africa North (southafricanorth), and data residency means we cannot...

  • 112 Views
  • 1 replies
  • 0 kudos
Latest Reply
arthurfr23
New Contributor III
  • 0 kudos

You're right, southafricanorth isn't currently on the Zerobus Ingest regional availability list for Azure. The Zerobus endpoint always runs in your workspace's region, so you can't point clients at an endpoint in another region and still write to you...

  • 0 kudos
inaafina
by • New Contributor
  • 149 Views
  • 2 replies
  • 0 kudos

Overwrite Job Errors

I keep on getting this error message when trying to create a parquet in my notebook. Has anyone found this error too? My overwrite path was working perfectly fine before.SparkConnectGrpcException: (java.lang.IllegalStateException) operationId: d5871d...

  • 149 Views
  • 2 replies
  • 0 kudos
Latest Reply
saurabh18cs
Honored Contributor III
  • 0 kudos

Hi  @inaafina setup like this , this is a new way to automatically resolve right package versions based on your compute. you have to use databricks cli  e.g. databricks environments setup-local \ --serverless-version 5 \ --dry-run

  • 0 kudos
1 More Replies
DB1To3
by • Valued Contributor II
  • 565 Views
  • 10 replies
  • 6 kudos

Resolved! Missing spark structured streaming metadata blobs?

I'm exploring spark structured streaming, and contemplating uses for it.I'd like to launch a job on a recurring basis from an external scheduler.  It will launch a spark job with trigger (availableNow=True). I'm concerned about overlapping executions...

DB1To3_0-1794341533309.png
  • 565 Views
  • 10 replies
  • 6 kudos
Latest Reply
saurabh18cs
Honored Contributor III
  • 6 kudos

Hi @DB1To3 Just make this concurrent run to 1 for your streaming jobs are your problem is sorted , it wont corrupt checkpoint locatiion  

  • 6 kudos
9 More Replies
DB1To3
by • Valued Contributor II
  • 214 Views
  • 4 replies
  • 1 kudos

Resolved! Statestore is not valid when using Spark Structured Streaming

I'm using spark structured streaming and it works properly at first.  But when I try to audit the checkpointLocation I can't get it to work.  It seems like it should be a relatively simple task to query the statestore and audit the processing of a st...

DB1To3_0-1794251574660.png
  • 214 Views
  • 4 replies
  • 1 kudos
Latest Reply
SajeevPS
New Contributor II
  • 1 kudos

I think the error is expected for this pipeline. The state reader functions (read_statestore / read_state_metadata) are for stateful Structured Streaming queries, meaning ones with a stateful operator such as aggregations, dropDuplicates, stream-stre...

  • 1 kudos
3 More Replies
rajkumar8k
by • New Contributor III
  • 203 Views
  • 2 replies
  • 6 kudos

Databricks Latest Features: Genie Code CLI, App Builder, Agent Memory and DABs Deployment History

Hi everyone,Databricks has shipped a lot of new features recently. Here’s a quick summary of the most useful ones, grouped by area.1. DABs Deployment HistoryWhat it is:Databricks Asset Bundles (DABs) now keep a history of every deployment. Each recor...

  • 203 Views
  • 2 replies
  • 6 kudos
Latest Reply
Jaswanth_alla
New Contributor
  • 6 kudos

Great summary! The features that stand out most to me are DABs Deployment History and UC JDBC Connections. Both solve real operational challenges around governance, visibility, and maintainability. Exciting to see Databricks continuing to simplify pl...

  • 6 kudos
1 More Replies
AlexMorgan
by • New Contributor
  • 328 Views
  • 4 replies
  • 1 kudos

How do you test data quality in a Delta Lake pipeline?

I'm exploring practical approaches to validating data quality in Databricks pipelines.For pipelines that ingest data into Delta tables, what checks do you typically automate at each stage?I'm particularly interested in handling:Schema changes and sch...

  • 328 Views
  • 4 replies
  • 1 kudos
Latest Reply
rajkumar8k
New Contributor III
  • 1 kudos

In my experience a combination works best:DLT / Lakeflow expectations check data quality while the pipeline runs. Use warn, drop or fail depending on how serious the issue is, and send bad records to a quarantine table instead of losing them.PySpark ...

  • 1 kudos
3 More Replies
Labels