topic Re: select 1 query not finishing in Data Engineering

select 1 query not finishing

Bepposbeste1993 — Sat, 30 Nov 2024 22:41:18 GMT

Hello,

I have the issue that even a query like "select 1" is not finishing. The sql warehouse runs infinite. I have no idea where to look for any issues because in the SPARK UI I cant see any error.

What is intresting is that also allpurpose clusters (multi node) are also affected

%sql
select 1

wont finish.

If I choose a single node cluster the thing is finishing within 2 seconds.

Any help would be appreciated that leads to the solution

Re: select 1 query not finishing

Alberto_Umana — Mon, 02 Dec 2024 13:45:49 GMT

Hi @Bepposbeste1993,

Have you tried adding the full path to the table you are querying? catalog.schema.table?

Additionally when using multi node do you have any special spark setting? You can validate cluster driver logs under log4j to validate if any errors during query executing to obtain more details on the behavior.

Re: select 1 query not finishing

Bepposbeste1993 — Mon, 09 Dec 2024 08:11:12 GMT

We solved the issue with MSFT and Databricks Support.

So our infrastructure was VNET injected. Technically it is possible to change the subnet size eventhough its not supported according to documentation.

The result is then a randomized behaviour depending where Databricks puts the nodes "on the subnet". If its on the old range - all is fine and if its on the new range, you get the above mentioned behaviour.

Changing the subnet range back solved our issue.

Re: select 1 query not finishing

Alberto_Umana — Tue, 10 Dec 2024 19:46:44 GMT

Hi @Bepposbeste1993,

Do you have the case ID raised for this issue?

Re: select 1 query not finishing

JasonESmith — Tue, 21 Jan 2025 17:22:33 GMT

So what did you change the subnet range back to?