External location not accessble with job and general purpose cluster but works fine with serverless

[Anmol_Chauhan]

I m setting up a pipeline in databricks.

For data source I create an external location and add my pipeline run_as SP as read files and read metadata permission. When i trigger the pipeline with serverless cluster, i m able to read data from external location but when i use job cluster, my pipeline stuck at cluster initializing phase, cluster not started even after 1 hour. Then i tried my notebook with general purpose cluster, but processing stuck at reading external location cell of notebook.

I did not understand what i m missing. If there is an issue with access then how it works with Serverless? Please help if someone knows the solution.

Sample code i m using for reading:

df = spark.read.format("delta").load("abfss://container@storage.dfs.core.windows.net/path")

[piyush18]

Check few things
1.Is The network in which we have job cluster or all purpose cluster, is that restricted?
2.The more likely chaces are due to the storage credential access, serverless takes the role of the SP having access to the location, while job cluster defaults to the user and since the user is not having access to the external location, thats why its failing

[balajij8 · Accepted solution]

@Anmol_Chauhan 

You have few distinct problems across the classic compute - cluster networking during boot and the cluster's compute access mode. Serverless works properly because Databricks manages the control plane routing and networking boundaries natively.

If you have custom VNet/VPC setups, Private Link endpoints or restrictive firewall/DNS configurations, the cluster nodes might be blocked from reaching the required Databricks control plane endpoints.

General-purpose cluster hanging on the read cell is likely an incompatible Access Mode issue. It works if the cluster is configured with Shared (User Isolation) or Single User access mode with proper access to the storage for users. If the cluster is running in No Isolation Shared mode (formerly Standard) or a legacy configuration without Unity Catalog enabled, Spark cannot resolve the cloud storage credentials and will hang indefinitely trying to authenticate.

Check your workspace network configuration, Private Link / DNS routing and network security groups to confirm node-to-control-plane communication is allowed. Confirm the workspace-level meta store storage configuration is valid and that the cluster policy or job definition is not injecting conflicting configurations. Edit the classic compute cluster settings and explicitly set the Access Mode to Shared or Single User and check again.