ERROR Description :

java.lang.Exception: Cannot run program "/local_disk0/pythonVirtualEnvDirs/virtualEnv-5acc1ea9-d03f-4de3-b76b-203d42614000/bin/python" (in directory "."): error=2, No such file or directory

at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)

at org.apache.spark.util.Utils$.executeCommand(Utils.scala:1507)

at org.apache.spark.util.Utils$.executeAndGetOutput(Utils.scala:1533)

at org.apache.spark.util.Utils$.executePythonAndGetOutput(Utils.scala:1576)

at org.apache.spark.util.DatabricksUtils$.getPythonLibDirFromExec(DatabricksUtils.scala:631)

at org.apache.spark.util.DatabricksUtils$.getPythonLibDirFromVirtualenv(DatabricksUtils.scala:622)

at org.apache.spark.util.DatabricksUtils$.createSiteCustomizeScriptForVirtualenv(DatabricksUtils.scala:667)

at org.apache.spark.util.DatabricksUtils$.$anonfun$getOrCreatePythonVirtualEnvDir$1(DatabricksUtils.scala:723)

at org.apache.spark.util.DatabricksUtils$.withStripedLock(DatabricksUtils.scala:411)

at org.apache.spark.util.DatabricksUtils$.getOrCreatePythonVirtualEnvDir(DatabricksUtils.scala:708)

at com.databricks.backend.daemon.driver.DBUtilsPythonEnvManager.getPythonExecutable(DBUtilsPythonEnvManager.scala:27)

at com.databricks.backend.daemon.driver.PythonDriverLocal.launchPython(PythonDriverLocal.scala:315)

at com.databricks.backend.daemon.driver.PythonDriverLocal.startPython0(PythonDriverLocal.scala:956)

at com.databricks.backend.daemon.driver.PythonDriverLocal.startPython(PythonDriverLocal.scala:1009)

at com.databricks.backend.daemon.driver.PythonDriverLocal.<init>(PythonDriverLocal.scala:1032)

at com.databricks.backend.daemon.driver.PythonDriverWrapper.instantiateDriver(DriverWrapper.scala:697)

at com.databricks.backend.daemon.driver.DriverWrapper.setupRepl(DriverWrapper.scala:335)

at com.databricks.backend.daemon.driver.DriverWrapper.run(DriverWrapper.scala:224)

at java.lang.Thread.run(Thread.java:750)

Caused by: java.lang.Throwable: error=2, No such file or directory

at java.lang.UNIXProcess.forkAndExec(Native Method)

at java.lang.UNIXProcess.<init>(UNIXProcess.java:247)

at java.lang.ProcessImpl.start(ProcessImpl.java:134)

at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)

NOTE : All Python and %sh commands are giving this error and scala commands are running fine and i am able to see spark-scala logs in Log Analytics.

Someone please let me know what is wrong with Python Virtual enviroment and how can i resolve this error and run python commands in Databricks notebook successfully?

Thanks & Regards,

Akanksha