I am experiencing an error when trying to commit output and Databricks notebook to GitHub. The error message states, "Error creating or updating commit outputs config file."Could you please assist me in resolving this issue?
I'm exploring practical approaches to validating data quality in Databricks pipelines.For pipelines that ingest data into Delta tables, what checks do you typically automate at each stage?I'm particularly interested in handling:Schema changes and sch...
The right pattern is to treat validation as multi layered instead of a single gate. You can start with handling schema changes and schema evolution at ingestion. You can configure Auto Loader with a dedicated schema at the ingestion point to prevent ...
Hello everyone,Around 3PM EST yesterday (2026-11-12), dozens of Lakeflow jobs started failing on both classic and serverless compute with distinct symptoms1. On classic compute (DBR 15.4), reading a file from S3 would result in a PATH NOT FOUND error...
Hello @ifishbein, I dug in and here is what I found. The serverless behavior is explainable, and your workaround is the right one. Serverless runs on Spark Connect, and in the Spark Connect flavor of PySpark SparkSession.getActiveSession() is backed ...
We need to bring the json data from external API and need to load it into lakebase API tables.Please suggest in connecting this.
Hi @Jack17,One small addition to the service principal advice above, because it makes testing a lot easier. Since 12 November, OAuth on-behalf-of for service principals is generally available, so you can test the Data API locally as the same service ...
Hello everyoneI'm trying to setup an ingestion pipeline from a mysql server and keep running into an issue. The setup was validated and the pipeline created. When I run the pipeline the destination tables are created in the schema, but then the follo...
Hi @MichaelAnckaert,The interesting part is the path in the error: s3:/ with nothing after it. That isn't coming from MySQL. It looks like the pipeline tried to resolve a storage location in Unity Catalog and got back an empty S3 path, and since the ...
Hi everyone,I am looking for a method to connect Databricks to Snowflake so I can query and transform the data directly in a notebook. I specifically want to avoid methods that just perform a one-time copy of the tables into Databricks, as I need the...
@raj , if you are happy with the answer and it addresses your issue please mark "Accept as Solution" so that others can have confidence in the response. Cheers, Lou.
Hi all,A full refresh of one Auto Loader streaming table in a serverless #Lakeflow (Declarative) pipeline has become about 4x slower (all to do with a single streaming table), and I'd like help understanding why. Nothing on our side changed. The onl...
Hi @rjmcbride what if you fiddle around wit max worksers for your serverless here
Hello Databricks Community,I am using Azure Databricks Lakeflow Connect with a PostgreSQL source.The CDC option is disabled in the PostgreSQL ingestion wizard, and I see the following message:This mode is not available in this workspace for the selec...
Hi @Pravieen worth looking this page : https://www.databricks.com/blog/push-based-cdc-unity-catalog-zerobus-ingest-and-debeziumBr
Lately we started using Direct Lake to use the results of our ingestion pipelines (gold tables) in PBI dashboards. This is working ok but has one annoying and troublesome issue. Which is: the gold tables are the end result of declarative pipelines so...
Hi @dbernstein_tp,Thanks for testing all of that and reporting back. Good to know deletion vectors didn't upset Direct Lake.I think your two findings are really the same problem. If most of the gold MVs are fully recomputed on every update, each refr...
Hi everyone,We ingest Oracle tables into a Bronze layer with Lakeflow Connect, and our Silver layer is a Lakeflow Spark Declarative Pipeline made of materialized views that read the Bronze tables and apply types and comments from our data dictionary....
@pradeep_singhThanks for the explanation, that makes sense. Let me give a bit more context on my case, because I think it's slightly different.The data has already been corrected in Oracle. The problem is that Lakeflow Connect resumes ingestion from ...
Hi Databricks Team,I'm trying to connect an AWS S3 bucket to Databricks Unity Catalog using an IAM role in Databricks Free Edition.EnvironmentDatabricks Edition: Free EditionCloud: AWSWorkspace region: us-east-2Metastore region: us-east-2S3 bucket re...
Hi @Visheshd_21 I did some digging.Same error here on Free Edition: Assume Role fails, and the other two checks are skipped. I went through all the suggestions above. The validate messages show the correct role ARN, UC principal and external ID; both...
Hi,Is there a plan to make lakebase as destination of Zerobus?Thanks.
Why assume 100 indexes on a Lakebase landing table while picturing a Delta table with at most 4 LC keys? Nobody will create 100 indexes for a landing table.Five seconds is far from a good num for some use cases. With Lakebase, data can be queried as ...
A pipeline change can run successfully and still ship wrong data: a join that duplicates rows, a filter that moves a revenue total by a few percent. With AI agents now writing more pipeline code, I wanted a check that runs before a change reaches pro...
The full-table checksum catching the 1-row delete is a nice find, sampling hiding it is exactly what people don't expect. One thing I'd double check is the sum of xxhash64. The hash spreads across the whole BIGINT range, so adding up a couple million...
I'm exploring spark structured streaming, and contemplating uses for it.I'd like to launch a job on a recurring basis from an external scheduler. It will launch a spark job with trigger (availableNow=True). I'm concerned about overlapping executions...
Hi @DB1To3 Just make this concurrent run to 1 for your streaming jobs are your problem is sorted , it wont corrupt checkpoint locatiion
Hello Databricks Community,I'm representing ASF CORPORATION, a piling, foundation engineering and construction company established in 2017 and based in Dhaka, Bangladesh.Our company specializes in hydraulic rotary piling, rotary drilling, auger borin...
Hi @asfcorporation, and welcome to the Databricks Community!Beyond the usual Lakeflow + Unity Catalog + AI/BI Dashboards stack, I’d suggest not inventing your own schema: geotechnical data already has open standards. AGS4 (UK) and DIGGS (US, by ASCE ...
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 586 | |
| 349 | |
| 287 |
