topic Re: DBT job stuck when running on databricks in Administration & Architecture

DBT job stuck when running on databricks

gilShin — Sun, 29 Oct 2023 13:48:08 GMT

Hi

I'm trying to run a DBT job on a databricks instance. The query should be run on the same instance.
When I run the job, I get to: Opening a new connection, currently in state init

It is stuck in that phase for a long time. I'm using IP access list which might be the issue and if it really is the issue, why can't I connect to localhost (or something like that)? If I can't use local host and must use a proper profile.yml DBT file, how can I add databricks list of IPs in eastus to the IP access list?

Thanks
Gil

Re: DBT job stuck when running on databricks

gilShin — Sun, 29 Oct 2023 14:02:51 GMT

OK. Added 0.0.0.0/0 to the IP access list but still the same result. This wasn't the issue (at least till I'll prove otherwise)

Re: DBT job stuck when running on databricks

gilShin — Sun, 29 Oct 2023 14:24:25 GMT

Another important point. The starter warehouse fails to start. After a few minutes the following issue appears:
Cluster Start-up Delayed. Please wait while we continue to try and start the cluster. No action is required from you.

We've increased cores quota today but I'm still facing this issue. I assume that this is the core of the issue.

Re: DBT job stuck when running on databricks

gilShin — Mon, 30 Oct 2023 08:14:17 GMT

Actually now I have a much bigger issue. My ISP has changed my address and I cannot access the databricks (or change the IP access list). Meaning, that was not a good solution to block public access to my worksapce (which I worked on for 3 days and couldn't find any other solution).
At first I have used Azure databricks standard tier and whatever I have tried, I couldn't block public access to my workspace (network security groups didn't help) so I have chnaged the tier from standard to premium and, apparently, that didn't help as well.
I'm really out of ideas here 😞

Re: DBT job stuck when running on databricks

gilShin — Wed, 01 Nov 2023 07:37:57 GMT

I recreated the databricks (there's no other way to solve that). If it was a production databricks workspace it was a disaster!
I have created a VM with static public IP and added this IP to the IP access list. Hopefully it'll become the last resort in case all other IPs will be replaced.
The above is an ugly workarround. No matter where I have looked, the only solution is to have private endpoint which requires other resources (peered VET + VPN) that I do not want to get into in this project.

As for the original issue, the issue, as mentioned above, was the SQL warehouse that couldn't start. I have created a new serverless SQL warehouse and added it to the job instead of the starter warehouse that was available and problem solved