Unable to open a file in dbfs. Trying to move files from Google Bucket to Azure Blob Storage

[editter]

Background:

I am attempting to download the google cloud sdk on Databricks. The end goal is to be able to use the sdk to transfer files from a Google Cloud Bucket to Azure Blob Storage using Databricks. (If you have any other ideas for this transfer please feel free to share. I do not want to use Azure Data Factory.)

I also have Unity Catalog enabled if that makes a difference.

Right now, I was first attempting to unzip the google cloud sdk in dbfs after I moved it to the following location. I know the file exists here:

%fs
ls dbfs:/tmp/google_sdk

Returns:
dbfs:/tmp/google_sdk/google_cloud_sdk_352_0_0_linux_x86_64_tar.gz

I have tried the following to open the file with tarfile. None have worked:

tar = tarfile.open('dbfs:/tmp/google_sdk/google_cloud_sdk_352_0_0_linux_x86_64_tar.gz', mode="r|gz")

tar = tarfile.open('/dbfs/tmp/google_sdk/google_cloud_sdk_352_0_0_linux_x86_64_tar.gz', mode="r|gz")

tar = tarfile.open('/tmp/google_sdk/google_cloud_sdk_352_0_0_linux_x86_64_tar.gz', mode="r|gz")

tar = tarfile.open('/dbfs/dbfs/tmp/google_sdk/google_cloud_sdk_352_0_0_linux_x86_64_tar.gz', mode="r|gz")

tar = tarfile.open('dbfs/tmp/google_sdk/google_cloud_sdk_352_0_0_linux_x86_64_tar.gz', mode="r|gz")

tar = tarfile.open('tmp/google_sdk/google_cloud_sdk_352_0_0_linux_x86_64_tar.gz', mode="r|gz")

All of them returning that no such file or directory exists, but I know it does. What am I missing here? Why am I not able to open this file?

Thanks for any help!

[editter]

Thanks you for the response!

2 Questions:

1. How would you create a cluster with the custom requirements for the google cloud sdk? Is that still possible for a Unity Catalog enabled cluster with Shared Access Mode?

2. Is a script action the same as a cluster init script? I couldn't find any documentation for script actions. 

I tried running that script on an existing cluster and it returned an AttributeError with no description. Just points to the line running the dbutils.cluster.submit_run (which I also can't find documenation for this command). I verified the cluster_id and driver_node_type_id were correct. 

Thanks for any help