Using Init scripts using DBX

naga_databricks
Contributor

I specify init scripts in my deployment.conf, as below:

  basic-static-cluster: &basic-static-cluster
    new_cluster:
      spark_version: "13.0.x-scala2.12"
      num_workers: 1
      node_type_id: "n2-highmem-2"
      init_scripts:
        - workspace:
            destination: file:fuse://conf/init_libraries.sh

On the job cluster initialization, i get an error:

Run result unavailable: run failed with error message
 Unexpected failure while waiting for the cluster (0509-103955-fc4levri) to be ready: Cluster 0509-103955-fc4levri is in unexpected state Terminating: INIT_SCRIPT_FAILURE(CLIENT_ERROR): databricks_error_message:RESOURCE_DOES_NOT_EXIST: No file found at /dbfs/dbx/mosaic_workflows/adb789dd822f44a0bf8ee77ac54fad7a/artifacts/conf/init_libraries.sh.

I see it uses the correct path where the file is present. I checked on the DBFS location and i can see this file.

Now, I am aware that the DBFS is going to be deprecated soon and I should maintain the file on the workspace files. How do I i keep the file on workspace using DBX? When storing the file on workspace, how do i reference this using FUSE?