04-06-2023 04:58 AM
ERROR Description :
java.lang.Exception: Cannot run program "/local_disk0/pythonVirtualEnvDirs/virtualEnv-5acc1ea9-d03f-4de3-b76b-203d42614000/bin/python" (in directory "."): error=2, No such file or directory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
at org.apache.spark.util.Utils$.executeCommand(Utils.scala:1507)
at org.apache.spark.util.Utils$.executeAndGetOutput(Utils.scala:1533)
at org.apache.spark.util.Utils$.executePythonAndGetOutput(Utils.scala:1576)
at org.apache.spark.util.DatabricksUtils$.getPythonLibDirFromExec(DatabricksUtils.scala:631)
at org.apache.spark.util.DatabricksUtils$.getPythonLibDirFromVirtualenv(DatabricksUtils.scala:622)
at org.apache.spark.util.DatabricksUtils$.createSiteCustomizeScriptForVirtualenv(DatabricksUtils.scala:667)
at org.apache.spark.util.DatabricksUtils$.$anonfun$getOrCreatePythonVirtualEnvDir$1(DatabricksUtils.scala:723)
at org.apache.spark.util.DatabricksUtils$.withStripedLock(DatabricksUtils.scala:411)
at org.apache.spark.util.DatabricksUtils$.getOrCreatePythonVirtualEnvDir(DatabricksUtils.scala:708)
at com.databricks.backend.daemon.driver.DBUtilsPythonEnvManager.getPythonExecutable(DBUtilsPythonEnvManager.scala:27)
at com.databricks.backend.daemon.driver.PythonDriverLocal.launchPython(PythonDriverLocal.scala:315)
at com.databricks.backend.daemon.driver.PythonDriverLocal.startPython0(PythonDriverLocal.scala:956)
at com.databricks.backend.daemon.driver.PythonDriverLocal.startPython(PythonDriverLocal.scala:1009)
at com.databricks.backend.daemon.driver.PythonDriverLocal.<init>(PythonDriverLocal.scala:1032)
at com.databricks.backend.daemon.driver.PythonDriverWrapper.instantiateDriver(DriverWrapper.scala:697)
at com.databricks.backend.daemon.driver.DriverWrapper.setupRepl(DriverWrapper.scala:335)
at com.databricks.backend.daemon.driver.DriverWrapper.run(DriverWrapper.scala:224)
at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.Throwable: error=2, No such file or directory
at java.lang.UNIXProcess.forkAndExec(Native Method)
at java.lang.UNIXProcess.<init>(UNIXProcess.java:247)
at java.lang.ProcessImpl.start(ProcessImpl.java:134)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
NOTE : All Python and %sh commands are giving this error and scala commands are running fine and i am able to see spark-scala logs in Log Analytics.
Someone please let me know what is wrong with Python Virtual enviroment and how can i resolve this error and run python commands in Databricks notebook successfully?
Thanks & Regards,
Akanksha
04-09-2023 09:33 PM
Hi Everyone, Looking for urgent solution to the above mentioned problem. Please help if anyone has clues about the above error and it's resolution.
04-18-2023 03:04 AM
@Akanksha Gupta :
The error message suggests that the Python executable file specified in the configuration of the Databricks cluster cannot be found or accessed. Specifically, it seems that the Python executable file at the path "/local_disk0/pythonVirtualEnvDirs/virtualEnv-5acc1ea9-d03f-4de3-b76b-203d42614000/bin/python" is missing or inaccessible.
To resolve this issue, you may try the following steps:
Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections.
Click here to register and join today!
Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.