Databricks Community

data_pulse

Hello Community.We recently enabled the following option on several existing Auto Loader streams:"cloudFiles.useManagedFileEvents": "true"Most streams continued working, but one now fails while restoring its existing checkpoint, referencing a file si...

data_pulse

I am looking at a scenario for team-level cost attribution using system.billing.usage and have a tagging gap for Databricks-managed services such as Predictive Optimization and Data Quality Monitoring.Tried adding a team/ownership tag directly to the...

data_pulse

@Khasim_1 I tested this with a small AUTO CDC pipeline and SEQUENCE BY STRUCT() works as expected for same-timestamp ties.Eg: In below dataEvent_ts seq_id operation 12:00 seq=1 INSERT 12:10 seq=1 UPDATE 12:10 seq=2 DELETEwith sequence...

data_pulse

After further investigation into it, have understood the root cause and the fix for it.After enabling Auto Loader managed file events across the platform: .option("cloudFiles.useManagedFileEvents", "true")the failed stream started discovering RocksDB...

data_pulse

Agree with the explanation from @ShamenParis. Databricks documents this explicitly too for ROW_NUMBER(): If the ordering is not unique, the result is non-deterministic.So if two customer_parent_name values have the same cnt, either one can receive ro...

data_pulse

Did you forgot to share the short note? If you are referring to running optimize after Merge, its worth mentioning that it doesn't need to be run after every Merge.MERGE can create or rewrite files, but whether OPTIMIZE is needed immediately depends...

data_pulse

@ShamenParis Thanks for looking into this. I have tried listOnStart/ validateOptions but unfortunately it didn't solve the issue. The error we have got (FAILED_READ_FILE.NO_HINT on the rocksdb zip) turned out to be different failure than continuation...

Databricks Community

User Stats

User Activity

Auto Loader stream fails on RocksDB checkpoint after enabling managed file events

Do UC table tags propagate to billing for Predictive Optimization and Data Quality Monitoring?

Re: How does AUTO CDC resolve same-timestamp tie in SEQUENCE BY?

Re: Auto Loader stream fails on RocksDB checkpoint after enabling managed file events

Re: Non-deterministic ROW_NUMBER() results across Unity Catalog environments

Re: Getting started with Delta Lake basics

Re: Auto Loader stream fails on RocksDB checkpoint after enabling managed file events