topic Re: Run stored bash in Databricks with %sh in Data Engineering

Run stored bash in Databricks with %sh

keunsoop — Tue, 03 Dec 2019 19:53:45 GMT

Hi, I made bash file in databricks and I can see that the file is stored as the following picture.

I was supposed to run this bash file through %sh cell, but as you see the following picture, I could not find bash file, which I could find through dbutils.fs.

Is there any way to run bash file stored in dbutils through the cell? This bash file is for installing packages, and executed as restarting cluster, but I would like to run this one manually so that I do not need to restart cluster whenever package is updated. Thanks.

Re: Run stored bash in Databricks with %sh

User16857282152 — Thu, 05 Dec 2019 15:15:34 GMT

The %sh command runs on the driver,

The driver has dbfs: mounted under /dbfs

So paths you might think of as dbfs:/FileStore end up being /dbfs/FileStore

I was able to execute a shell script by uploading to the FileStore

Moving to current working directory with a

%sh mv command

and then executing with a

%sh sh myscript.sh

Re: Run stored bash in Databricks with %sh

Lokesh_Vish — Wed, 22 Jul 2020 13:00:03 GMT

Hello @Tom Hanlon

Could you please explain in details how it is done,

I am not able to run the .sh file in databricks

Re: Run stored bash in Databricks with %sh

jimnaik — Wed, 09 Feb 2022 18:55:44 GMT

@Tom Hanlon Uploading to FileStore path is to create data. How can we easily access the directory and upload .sh file to run the same in the workspace?

Re: Run stored bash in Databricks with %sh

jimnaik — Wed, 09 Feb 2022 18:58:15 GMT

@Lokesh_Vish were you able to execute it?

I am facing similar issue and not able to upload and execute .sh as well as .py file.

Re: Run stored bash in Databricks with %sh

Anonymous — Thu, 10 Feb 2022 15:15:58 GMT

Hello @Jimit Naik - My name is Piper, and I'm a Databricks community moderator. Thank you for asking about this. We'll give the community a chance to respond first. 🙂

Re: Run stored bash in Databricks with %sh

Atanu — Tue, 08 Mar 2022 13:59:30 GMT

this can be run through init script I think,@keunsoop

Re: Run stored bash in Databricks with %sh

jose_gonzalez — Mon, 11 Apr 2022 18:48:35 GMT

Hi @keunsoop ,

Are you able to run your code using an init script? I would like to share some docs in case you might have some questions https://docs.databricks.com/clusters/init-scripts.html