ERROR Description :
java.lang.Exception: Cannot run program "/local_disk0/pythonVirtualEnvDirs/virtualEnv-5acc1ea9-d03f-4de3-b76b-203d42614000/bin/python" (in directory "."): error=2, No such file or directory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
at org.apache.spark.util.Utils$.executeCommand(Utils.scala:1507)
at org.apache.spark.util.Utils$.executeAndGetOutput(Utils.scala:1533)
at org.apache.spark.util.Utils$.executePythonAndGetOutput(Utils.scala:1576)
at org.apache.spark.util.DatabricksUtils$.getPythonLibDirFromExec(DatabricksUtils.scala:631)
at org.apache.spark.util.DatabricksUtils$.getPythonLibDirFromVirtualenv(DatabricksUtils.scala:622)
at org.apache.spark.util.DatabricksUtils$.createSiteCustomizeScriptForVirtualenv(DatabricksUtils.scala:667)
at org.apache.spark.util.DatabricksUtils$.$anonfun$getOrCreatePythonVirtualEnvDir$1(DatabricksUtils.scala:723)
at org.apache.spark.util.DatabricksUtils$.withStripedLock(DatabricksUtils.scala:411)
at org.apache.spark.util.DatabricksUtils$.getOrCreatePythonVirtualEnvDir(DatabricksUtils.scala:708)
at com.databricks.backend.daemon.driver.DBUtilsPythonEnvManager.getPythonExecutable(DBUtilsPythonEnvManager.scala:27)
at com.databricks.backend.daemon.driver.PythonDriverLocal.launchPython(PythonDriverLocal.scala:315)
at com.databricks.backend.daemon.driver.PythonDriverLocal.startPython0(PythonDriverLocal.scala:956)
at com.databricks.backend.daemon.driver.PythonDriverLocal.startPython(PythonDriverLocal.scala:1009)
at com.databricks.backend.daemon.driver.PythonDriverLocal.<init>(PythonDriverLocal.scala:1032)
at com.databricks.backend.daemon.driver.PythonDriverWrapper.instantiateDriver(DriverWrapper.scala:697)
at com.databricks.backend.daemon.driver.DriverWrapper.setupRepl(DriverWrapper.scala:335)
at com.databricks.backend.daemon.driver.DriverWrapper.run(DriverWrapper.scala:224)
at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.Throwable: error=2, No such file or directory
at java.lang.UNIXProcess.forkAndExec(Native Method)
at java.lang.UNIXProcess.<init>(UNIXProcess.java:247)
at java.lang.ProcessImpl.start(ProcessImpl.java:134)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
NOTE : All Python and %sh commands are giving this error and scala commands are running fine and i am able to see spark-scala logs in Log Analytics.
Someone please let me know what is wrong with Python Virtual enviroment and how can i resolve this error and run python commands in Databricks notebook successfully?
Thanks & Regards,
Akanksha