Re: Run stored bash in Databricks with %sh

keunsoop · ‎12-03-2019

Hi, I made bash file in databricks and I can see that the file is stored as the following picture.

I was supposed to run this bash file through %sh cell, but as you see the following picture, I could not find bash file, which I could find through dbutils.fs.

Is there any way to run bash file stored in dbutils through the cell? This bash file is for installing packages, and executed as restarting cluster, but I would like to run this one manually so that I do not need to restart cluster whenever package is updated. Thanks.

User16857282152 · ‎12-05-2019

The %sh command runs on the driver,

The driver has dbfs: mounted under /dbfs

So paths you might think of as dbfs:/FileStore end up being /dbfs/FileStore

I was able to execute a shell script by uploading to the FileStore

Moving to current working directory with a

%sh mv command

and then executing with a

%sh sh myscript.sh

jimnaik · ‎02-09-2022

@Tom Hanlon Uploading to FileStore path is to create data. How can we easily access the directory and upload .sh file to run the same in the workspace?

Lokesh_Vish · ‎07-22-2020

Hello @Tom Hanlon

Could you please explain in details how it is done,

I am not able to run the .sh file in databricks

jimnaik · ‎02-09-2022

@Lokesh_Vish were you able to execute it?

I am facing similar issue and not able to upload and execute .sh as well as .py file.

Anonymous · ‎02-10-2022

Hello @Jimit Naik - My name is Piper, and I'm a Databricks community moderator. Thank you for asking about this. We'll give the community a chance to respond first. 🙂

Atanu · ‎03-08-2022

this can be run through init script I think,@keunsoop

jose_gonzalez · ‎04-11-2022

Hi @keunsoop ,

Are you able to run your code using an init script? I would like to share some docs in case you might have some questions https://docs.databricks.com/clusters/init-scripts.html

View solution in original post