DBT job stuck when running on databricks

[gilShin]

Hi

I'm trying to run a DBT job on a databricks instance. The query should be run on the same instance.
When I run the job, I get to: Opening a new connection, currently in state init

It is stuck in that phase for a long time. I'm using IP access list which might be the issue and if it really is the issue, why can't I connect to localhost (or something like that)? If I can't use local host and must use a proper profile.yml DBT file, how can I add databricks list of IPs in eastus to the IP access list?

Thanks
Gil

[gilShin]

OK. Added 0.0.0.0/0 to the IP access list but still the same result. This wasn't the issue (at least till I'll prove otherwise)

[gilShin]

Another important point. The starter warehouse fails to start. After a few minutes the following issue appears:
Cluster Start-up Delayed. Please wait while we continue to try and start the cluster. No action is required from you.

We've increased cores quota today but I'm still facing this issue. I assume that this is the core of the issue.

[Kaniz]

Hi @gilShin , 

If you're using Azure Databricks and facing issues with opening a new connection when running DBT jobs, you can try the following troubleshooting steps:

1. Check the IP Access List settings:

First, verify that the IP Access List settings in the Databricks workspace are correctly configured. Check that the IP addresses or ranges being used either in the IP Access List or in the network security groups are valid. Ensure you have added the IP range of the machine or network where the DBT jobs run to the IP Access List.

2. Verify the network connectivity:

Ensure that the network settings allow the machine running DBT jobs to connect to the Azure Databricks instance. Check if your IT/Operations team has imposed any rules or restrictions on the internet connection, which could block the traffic from the client machine.

3. Try adding the IP range manually:

Suppose the IP Access List is not being updated correctly. In that case, you can add the IP addresses manually by navigating to the Workspace's Security tab, selecting IP Access Lists on the left-hand side, and then clicking the "Add" button to add the IP range manually.

4. Use a proper DBT profile file:

Ensure that you're using a valid and properly configured DBT profile file with the correct environment, cluster details, authentication parameters, and other configurations specific to the Azure Databricks (e.g., host, token, org_id).

5. Use JDBC as the connector:

Try using the JDBC connector instead of the ODBC connector to connect to Databricks to see if that makes any difference. Power BI desktop can use the JDBC connector, which can be found in the Databricks account settings under the "JDBC/ODBC" tab.

These steps should help you resolve the issues with opening a new connection when running DBT jobs on Azure Databricks.

[gilShin]

Actually now I have a much bigger issue. My ISP has changed my address and I cannot access the databricks (or change the IP access list). Meaning, that was not a good solution to block public access to my worksapce (which I worked on for 3 days and couldn't find any other solution).
At first I have used Azure databricks standard tier and whatever I have tried, I couldn't block public access to my workspace (network security groups didn't help) so I have chnaged the tier from standard to premium and, apparently, that didn't help as well. 
I'm really out of ideas here 😞 

[gilShin · Accepted solution]

I recreated the databricks (there's no other way to solve that). If it was a production databricks workspace it was a disaster!
I have created a VM with static public IP and added this IP to the IP access list. Hopefully it'll become the last resort in case all other IPs will be replaced.
The above is an ugly workarround. No matter where I have looked, the only solution is to have private endpoint which requires other resources (peered VET + VPN) that I do not want to get into in this project.

As for the original issue, the issue, as mentioned above, was the SQL warehouse that couldn't start. I have created a new serverless SQL warehouse and added it to the job instead of the starter warehouse that was available and problem solved