cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

How to set the ABFSS URL for Azure Databricks Init Scripts that have spaces in directory names?

FRG96
New Contributor III

I want to use an Init Script on ADLS Gen2 location for my Azure Databricks 11.3 and 12.2 clusters.

The init_script.sh is placed in a directory that has spaces in it:

https://storageaccount1.blob.core.windows.net/container1/directory%20with%20spaces/subdirectory%20wi...

When I pass the equivalent ABFSS URL to Databricks:

abfss://container1@storageaccount1.dfs.core.windows.net/directory%20with%20spaces/subdirectory%20with%20spaces/init_script.sh

The ABFSS URL gets URL encoded again and %20 is converted to %2520 as seen in the cluster creation failure event log:

{
  "reason": {
    "code": "INIT_SCRIPT_FAILURE",
    "type": "CLIENT_ERROR",
    "parameters": {
      "instance_id": "a2dcf9efabaf4fd6b3ba3409b09a567f",
      "databricks_error_message": "Cluster scoped init script abfss://container1@storageaccount1.dfs.core.windows.net/directory%20with%20spaces/subdirectory%20with%20spaces/init_script.sh failed: Timed out with exception after 5 attempts (debugStr = 'Reading remote file for init script'), Caused by: java.io.FileNotFoundException: Operation failed: \"The specified path does not exist.\", 404, HEAD, https://storageaccount1.dfs.core.windows.net/container1/directory%2520with%2520spaces/subdirectory%2...;
    }
  }
}

Also, when I try to pass spaces in ABFSS URL like this:

abfss://container1@storageaccount1.dfs.core.windows.net/directory with spaces/subdirectory with spaces/init_script.sh

I get this error:

{
  "reason": {
    "code": "INIT_SCRIPT_FAILURE",
    "type": "CLIENT_ERROR",
    "parameters": {
      "instance_id": "501ad15cd1b1484e973fde156aa85b83",
      "databricks_error_message": "Cluster scoped init script abfss://container1@storageaccount1.dfs.core.windows.net/directory with spaces/subdirectory with spaces/init_script.sh failed: Illegal character in path at index 71: abfss://container1@storageaccount1.dfs.core.windows.net/directory with spaces/subdirectory with spaces/init_script.sh, Caused by: java.net.URISyntaxException: Illegal character in path at index 71: abfss://container1@storageaccount1.dfs.core.windows.net/directory with spaces/subdirectory with spaces/init_script.sh"
    }
  }
}

So, what's the correct way to pass the ABFSS URL when the directory has spaces?

0 REPLIES 0
Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.