How to run commands on the executor

brickster_2018 — Fri, 25 Jun 2021 19:17:59 GMT

Using %sh, I am able to run commands on the notebook and get output. How can i run a command on the executor and get the output. I want to avoid using the Spark API's

Re: How to run commands on the executor

brickster_2018 — Fri, 25 Jun 2021 19:19:42 GMT

It's not possible to use %sh to run commands on the executor. The below code can be used to run commands on the executor and get the output

var res=sc.runOnEachExecutor[String]({ () =>
import sys.process._
  
       var cmd_Result=Seq("bash", "-c", "hostname").!!
      cmd_Result
    }, 100.seconds)

topic Re: How to run commands on the executor in Data Engineering

How to run commands on the executor

Re: How to run commands on the executor