topic Re: Streaming Queries Failing Frequently in DBR 10.4 LTS for the Last Week in Data Engineering

Streaming Queries Failing Frequently in DBR 10.4 LTS for the Last Week

Soma — Tue, 09 Aug 2022 14:47:25 GMT

DBR 10.4 LTS is failing frequently due to GC overhead once in half an hour.

Can anyone from Databricks Team let me know if we have some existing tickets or bugs.

Note : We used the same configuration and same DBR for almost last 3 months.

When checking the logs there are issues in writing SST files to rocks db and few issues are there in connecting to azure ( timeout error )

Re: Streaming Queries Failing Frequently in DBR 10.4 LTS for the Last Week

Prabakar — Tue, 09 Aug 2022 15:26:32 GMT

if you are seeing timeout errors, was there any network change that happened recently on your side? Do you use default DNS or custom DNS? You can create an init script to capture the network traffic.

https://docs.microsoft.com/en-us/azure/databricks/kb/dev-tools/use-tcpdump-create-pcap-files

Re: Streaming Queries Failing Frequently in DBR 10.4 LTS for the Last Week

Soma — Tue, 09 Aug 2022 15:55:13 GMT

hi @Prabakar Ammeappin the timeout is very less ,

we see a lot of rocks db copy happening more than usual

Re: Streaming Queries Failing Frequently in DBR 10.4 LTS for the Last Week

Prabakar — Tue, 09 Aug 2022 16:03:15 GMT

hi @somanath Sankaran could you please add the stack message?

Re: Streaming Queries Failing Frequently in DBR 10.4 LTS for the Last Week

Vidula — Thu, 08 Sep 2022 10:14:21 GMT

Hi @somanath Sankaran

Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help.

We'd love to hear from you.

Thanks!

Re: Streaming Queries Failing Frequently in DBR 10.4 LTS for the Last Week

Soma — Mon, 26 Sep 2022 15:29:26 GMT

hi @Vidula Khanna have raised a support ticket to ADB from client side.

We can close this however it seems like DBR Version 11.2 and above has some fixes for the RocksDB memory leak based on communication with Databricks developer team