Run stored bash in Databricks with %sh

[keunsoop]

Hi, I made bash file in databricks and I can see that the file is stored as the following picture.

I was supposed to run this bash file through %sh cell, but as you see the following picture, I could not find bash file, which I could find through dbutils.fs.

Is there any way to run bash file stored in dbutils through the cell? This bash file is for installing packages, and executed as restarting cluster, but I would like to run this one manually so that I do not need to restart cluster whenever package is updated. Thanks.

[User16857282152]

The %sh command runs on the driver,

The driver has dbfs: mounted under /dbfs

So paths you might think of as dbfs:/FileStore end up being /dbfs/FileStore

I was able to execute a shell script by uploading to the FileStore

Moving to current working directory with a

%sh mv command

and then executing with a

%sh sh myscript.sh

[jimnaik]

@Tom Hanlon​  Uploading to FileStore path is to create data. How can we easily access the directory and upload .sh file to run the same in the workspace?

[Lokesh_Vish]

Hello @Tom Hanlon​ 

Could you please explain in details how it is done,

I am not able to run the .sh file in databricks

[jimnaik]

@Lokesh_Vish​ were you able to execute it?

I am facing similar issue and not able to upload and execute .sh as well as .py file.

[Anonymous]

Hello @Jimit Naik​ - My name is Piper, and I'm a Databricks community moderator. Thank you for asking about this. We'll give the community a chance to respond first. 🙂

[Kaniz]

Hi @keunsoop​ , @Lokeshkumar Madabathula​ and @Jimit Naik​ , Please try this after uploading it to the 'Data' section of the workspace:-

%sh /dbfs/***/***/***.sh

[Atanu]

this can be run through init script I think,@keunsoop​ 

[jose_gonzalez · Accepted solution]

Hi @keunsoop​ ,

Are you able to run your code using an init script? I would like to share some docs in case you might have some questions https://docs.databricks.com/clusters/init-scripts.html