Error While Running Workloads on Databricks Free Edition – Serverless Compute

[Bharat_Kumar_AV]

Hi Team,

I am using the Databricks Free Edition and I am facing an issue while running workloads using Serverless Compute.

When I try to run the workload, I receive the following error:

“An error occurred while trying to provision serverless compute. Please try again or contact support.”

I have tried running the workload again, but the issue persists.

Could you please check if there is any issue with the serverless compute provisioning for my workspace or if any configuration/action is required from my side?

[szymon_dybczak]

Hi @Bharat_Kumar_AV ,

The error is more likely caused by intermittent connectivity between the Lakeflow Connect gateway and SQL Server.

Lakeflow Connect SQL Server gateway runs continuously on Databricks classic compute. Therefore, the network path from the Databricks classic-compute VPC/VNet to SQL Server must remain continuously available.

I would check thefollowing:

1. Verify DNS resolution of the SQL Server hostname from the Databricks classic-compute network. If the hostname resolves to multiple IP addresses, confirm that all IP addresses are reachable.

2. Verify firewall/security-group/NSG rules for all Databricks subnets where the ingestion gateway compute can be created. Make sure TCP connectivity to the SQL Server port is consistently allowed.

3. Check VPN, VPC/VNet peering, PrivateLink/private endpoint, Direct Connect/ExpressRoute, NAT gateway, or other networking components between Databricks and SQL Server for intermittent failures.

4. Compare the exact gateway failure timestamps with SQL Server connection/error logs. If SQL Server does not see an incoming connection during the failure, the problem is most likely in DNS/networking. If SQL Server sees and rejects the connection, investigate SQL Server-side limits, TLS, authentication, or listener configuration.

5. If SQL Server uses a named instance or dynamic port, consider using a static TCP port and stable DNS name.

6. Is your SQL Server behind Availability Groups? Maybe AG failover happens?

[szymon_dybczak]

Sorry guys, I've replied in a wrong thread. Moderators, please remove that post 🙂

[data_pulse]

@Bharat_Kumar_AV 
I just tried Free edition brand new by signing up and able to run serverless without any issues. The error posted seems generic and worth checking below usual causes. 

• old serverless environment version (if notebook was imported) as Imported HTML notebooks stay pinned to old version and don't auto-upgrade.
• Open the notebook -> check Environment (At far right) -> Environment version -> switch to current one.
  Also try running same in brand new notebook and see if its the same issue.
• Free Edition is bound by quota, If you exceed your quota, your workspace's compute resources will be shut down and unavailable for the rest of the day.
• Free Edition also allows SQL warehouse (2x-small) and limited notebook compute. Try if the SQL warehouse is working first and also stop the SQL warehouse, retry the notebook compute.
• Check any status issues in your region using https://status.databricks.com.
• Databricks docs also mention serverless limitations, check this once if you hit any of notebook / job / streaming / Network / data access limitations.