Databricks Community

@thekushalvishwa AI can write the pipeline in few seconds. It might not explain the DBU bill to finance. A timely post, and one I suspect many of us relate to.My take is that AI has made writing the code cheap, so our value has moved to the parts it ...

The right pattern is to treat validation as multi layered instead of a single gate. You can start with handling schema changes and schema evolution at ingestion. You can configure Auto Loader with a dedicated schema at the ingestion point to prevent ...

Hi @Jack17,One small addition to the service principal advice above, because it makes testing a lot easier. Since 12 November, OAuth on-behalf-of for service principals is generally available, so you can test the Data API locally as the same service ...

Hi @MichaelAnckaert,The interesting part is the path in the error: s3:/ with nothing after it. That isn't coming from MySQL. It looks like the pipeline tried to resolve a storage location in Unity Catalog and got back an empty S3 path, and since the ...

@raj , if you are happy with the answer and it addresses your issue please mark "Accept as Solution" so that others can have confidence in the response. Cheers, Lou.

Hi @rjmcbride what if you fiddle around wit max worksers for your serverless here  

Hi @Pravieen worth looking this page : https://www.databricks.com/blog/push-based-cdc-unity-catalog-zerobus-ingest-and-debeziumBr

Hi @dbernstein_tp,Thanks for testing all of that and reporting back. Good to know deletion vectors didn't upset Direct Lake.I think your two findings are really the same problem. If most of the gold MVs are fully recomputed on every update, each refr...

@pradeep_singhThanks for the explanation, that makes sense. Let me give a bit more context on my case, because I think it's slightly different.The data has already been corrected in Oracle. The problem is that Lakeflow Connect resumes ingestion from ...

Hi @Visheshd_21 I did some digging.Same error here on Free Edition: Assume Role fails, and the other two checks are skipped. I went through all the suggestions above. The validate messages show the correct role ARN, UC principal and external ID; both...

Why assume 100 indexes on a Lakebase landing table while picturing a Delta table with at most 4 LC keys? Nobody will create 100 indexes for a landing table.Five seconds is far from a good num for some use cases. With Lakebase, data can be queried as ...

The full-table checksum catching the 1-row delete is a nice find, sampling hiding it is exactly what people don't expect. One thing I'd double check is the sum of xxhash64. The hash spreads across the whole BIGINT range, so adding up a couple million...