Databricks

YFL · 12-09-2021

Hi, I want to keep track of the streaming lag from the source table, which is a delta table. I see that in query progress logs, there is some information about the last version and the last file in the version for the end offset, but this don't give ...

YFL · 05-15-2022

Hey @Vartika Nain . The issue was resolved. Thanks.

YFL · 01-03-2022

Hi, @Piper Wilson . Those metrics don't solve the issue. What I am asking for is to keep tracking the lag between the running streaming and the source delta table. Those metrics give me a lot of information, like input rates (as bytes and as files) ...

YFL · 12-13-2021

I am.Yerachmiel Feltzman | Data Platform Developer

YFL · 12-12-2021

Hi, @Jose Gonzalez , I don't see in the link something that states the lag from the source delta table. Thanks anyway.

YFL · 12-10-2021

Thanks, Kaniz. This is a highly important question for some production jobs we have (and we are highly invested in Databricks and Delta). I have seen others through the internet asking the same question, as well.Thank you.Yerachmiel Feltzman | Data P...

Databricks

User Stats

User Activity

When delta is a streaming source, how can we get the consumer lag?

Re: When delta is a streaming source, how can we get the consumer lag?

Re: When delta is a streaming source, how can we get the consumer lag?

Re: When delta is a streaming source, how can we get the consumer lag?

Re: When delta is a streaming source, how can we get the consumer lag?

Re: When delta is a streaming source, how can we get the consumer lag?