I m setting up a pipeline in databricks.

For data source I create an external location and add my pipeline run_as SP as read files and read metadata permission. When i trigger the pipeline with serverless cluster, i m able to read data from external location but when i use job cluster, my pipeline stuck at cluster initializing phase, cluster not started even after 1 hour. Then i tried my notebook with general purpose cluster, but processing stuck at reading external location cell of notebook.

I did not understand what i m missing. If there is an issue with access then how it works with Serverless? Please help if someone knows the solution.

Sample code i m using for reading:

df = spark.read.format("delta").load("abfss://container@storage.dfs.core.windows.net/path")