I want to use an Init Script on ADLS Gen2 location for my Azure Databricks 11.3 and 12.2 clusters.
The init_script.sh is placed in a directory that has spaces in it:
https://storageaccount1.blob.core.windows.net/container1/directory%20with%20spaces/subdirectory%20wi...
When I pass the equivalent ABFSS URL to Databricks:
abfss://container1@storageaccount1.dfs.core.windows.net/directory%20with%20spaces/subdirectory%20with%20spaces/init_script.sh
The ABFSS URL gets URL encoded again and %20 is converted to %2520 as seen in the cluster creation failure event log:
{
"reason": {
"code": "INIT_SCRIPT_FAILURE",
"type": "CLIENT_ERROR",
"parameters": {
"instance_id": "a2dcf9efabaf4fd6b3ba3409b09a567f",
"databricks_error_message": "Cluster scoped init script abfss://container1@storageaccount1.dfs.core.windows.net/directory%20with%20spaces/subdirectory%20with%20spaces/init_script.sh failed: Timed out with exception after 5 attempts (debugStr = 'Reading remote file for init script'), Caused by: java.io.FileNotFoundException: Operation failed: \"The specified path does not exist.\", 404, HEAD, https://storageaccount1.dfs.core.windows.net/container1/directory%2520with%2520spaces/subdirectory%2...;
}
}
}
Also, when I try to pass spaces in ABFSS URL like this:
abfss://container1@storageaccount1.dfs.core.windows.net/directory with spaces/subdirectory with spaces/init_script.sh
I get this error:
{
"reason": {
"code": "INIT_SCRIPT_FAILURE",
"type": "CLIENT_ERROR",
"parameters": {
"instance_id": "501ad15cd1b1484e973fde156aa85b83",
"databricks_error_message": "Cluster scoped init script abfss://container1@storageaccount1.dfs.core.windows.net/directory with spaces/subdirectory with spaces/init_script.sh failed: Illegal character in path at index 71: abfss://container1@storageaccount1.dfs.core.windows.net/directory with spaces/subdirectory with spaces/init_script.sh, Caused by: java.net.URISyntaxException: Illegal character in path at index 71: abfss://container1@storageaccount1.dfs.core.windows.net/directory with spaces/subdirectory with spaces/init_script.sh"
}
}
}
So, what's the correct way to pass the ABFSS URL when the directory has spaces?