Databricks Community

Adding to the notebook approach above, two things to plan for. The Data API wants an OAuth bearer token on every request, and those tokens are short lived (about an hour per the docs), so a scheduled job should fetch a fresh one at the start of each ...

@pradeep_singhThanks for the explanation, that makes sense. Let me give a bit more context on my case, because I think it's slightly different.The data has already been corrected in Oracle. The problem is that Lakeflow Connect resumes ingestion from ...

Your metrics already narrow this down significantly. Per-file processing time (~21–24 ms), storage latency, retries, and tasks per batch are all unchanged, yet concurrent running tasks dropped from ~21–24 down to ~7–8 while the backlog stayed above 1...

What has worked for us is combining rule-based checks inside the pipeline with monitoring outside it.Checks we automate:Schema changes: Compare the incoming schema with the expected or last-known schema on every run, and alert on new, removed or rety...

Hi @Visheshd_21 I did some digging.Same error here on Free Edition: Assume Role fails, and the other two checks are skipped. I went through all the suggestions above. The validate messages show the correct role ARN, UC principal and external ID; both...

Why assume 100 indexes on a Lakebase landing table while picturing a Delta table with at most 4 LC keys? Nobody will create 100 indexes for a landing table.Five seconds is far from a good num for some use cases. With Lakebase, data can be queried as ...

The full-table checksum catching the 1-row delete is a nice find, sampling hiding it is exactly what people don't expect. One thing I'd double check is the sum of xxhash64. The hash spreads across the whole BIGINT range, so adding up a couple million...

Hello @Raj_DB, I took a look at both internal and external documentation and here is what I found. The thread already has the right answer: Lakehouse Federation. @saurabh18cs, @Harshad_B, @AbhilashNagilla, @Aravind_Reddy, @Jen_Evangelista and @balaji...

Hi @DB1To3 Just make this concurrent run to 1 for your streaming jobs are your problem is sorted , it wont corrupt checkpoint locatiion  

Hi @asfcorporation, and welcome to the Databricks Community!Beyond the usual Lakeflow + Unity Catalog + AI/BI Dashboards stack, I’d suggest not inventing your own schema: geotechnical data already has open standards. AGS4 (UK) and DIGGS (US, by ASCE ...

Hi @dbernstein_tp,Thanks for testing all of that and reporting back. Good to know deletion vectors didn't upset Direct Lake.I think your two findings are really the same problem. If most of the gold MVs are fully recomputed on every update, each refr...

Hi @krishgarikipati Could you clarify what specific issue you are facing with the current Parquet-based Delta implementation that makes you want to explore a different file format? Is the concern primarily the amount of data being rewritten for a ver...

Checked the full refresh.I can easily control pauses as we only update end user tables one a dayIt is SQL Server CDC. Full refresh completed Ok

This type of issue can sometimes occur when AWS IAM permissions are correctly configured, but Unity Catalog is unable to authorize access through the storage credential or external location. I would recommend checking a few things before changing the...