Databricks Community

YuriS · 01-26-2026

After implementing StreamingQueryListener to enable integration with our monitoring solution we have noticed some strange metrics for our DeltaSource streams (based on https://learn.microsoft.com/en-us/azure/databricks/structured-streaming/stream-mon...

YuriS · 10-07-2025

Let’s say we have big data application where data loss is not an option.Having GZRS (geo-zone-redundant storage) redundancy we would achieve zero data loss if primary region is alive – writer is waiting for acks from two or more Azure availability zo...

YuriS · 04-18-2025

Could someone please advise regarding VACUUM with Azure Storage Inventory Report as i have failed to make it work.DBR 15.4 LTS, VACUUM command is being run with USING INVENTORY clause, as follows:VACUUM schema.table USING INVENTORY ( select 'https://...

YuriS · 02-13-2026

Trigger interval is set to 1 minute; maxOffsetsPerTrigger is set to 20k. That's all.And once again when i am comparing either offsets (from the StreamingQueryListener events) or actual data in target table (batch_id is in target table, so it is very ...

YuriS · 02-12-2026

Could someone from Databricks answer on that question.On top of the above numInputRows metric is not (imho) reliable information as well.Checked several stateless streams (simple foreachBatch function with extraction and straightforward validation wh...

YuriS · 10-16-2025

@mark_ott Mark, thank you for so detailed response.We are currently evaluating two approaches for seamless, lossless failover with full integrity in stateless streaming scenarios. 1) Parse Databricks Offsets files and2) Enable row trackingthat we’ve...

YuriS · 04-22-2025

After additional investigation it turned out the proper "fully-qualified-URL" path should be'dbfs:/mnt/...''dbfs:/mnt/{endpoint}/' || ir.Name as path,and not 'https://xxx.blob.core.windows.net/' || ir.Name as path,

Databricks Community

User Stats

User Activity

StreamingQueryListener metrics strange behaviour (inputRowsPerSecond metric is set to 0)

How to reduce data loss for Delta Lake on Azure when failing from primary to secondary regions?

VACUUM with Azure Storage Inventory Report is not working

Re: StreamingQueryListener metrics strange behaviour (inputRowsPerSecond metric is set to 0)

Re: StreamingQueryListener metrics strange behaviour (inputRowsPerSecond metric is set to 0)

Re: How to reduce data loss for Delta Lake on Azure when failing from primary to secondary regions?

Re: VACUUM with Azure Storage Inventory Report is not working