How to run commands on the executor

brickster_2018 · ‎06-25-2021

Using %sh, I am able to run commands on the notebook and get output. How can i run a command on the executor and get the output. I want to avoid using the Spark API's

brickster_2018 · ‎06-25-2021

It's not possible to use %sh to run commands on the executor. The below code can be used to run commands on the executor and get the output

var res=sc.runOnEachExecutor[String]({ () =>
import sys.process._
  
       var cmd_Result=Seq("bash", "-c", "hostname").!!
      cmd_Result
    }, 100.seconds)

View solution in original post