ERROR : Failure starting repl. Try detaching and re-attaching the notebook. Getting this error when running any python command in 10.4LTS cluster configured with https://github.com/mspnp/spark-monitoring to send databricks spark logs to Log Analytics.

[akanksha_gupta]

ERROR Description :

java.lang.Exception: Cannot run program "/local_disk0/pythonVirtualEnvDirs/virtualEnv-5acc1ea9-d03f-4de3-b76b-203d42614000/bin/python" (in directory "."): error=2, No such file or directory

at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)

at org.apache.spark.util.Utils$.executeCommand(Utils.scala:1507)

at org.apache.spark.util.Utils$.executeAndGetOutput(Utils.scala:1533)

at org.apache.spark.util.Utils$.executePythonAndGetOutput(Utils.scala:1576)

at org.apache.spark.util.DatabricksUtils$.getPythonLibDirFromExec(DatabricksUtils.scala:631)

at org.apache.spark.util.DatabricksUtils$.getPythonLibDirFromVirtualenv(DatabricksUtils.scala:622)

at org.apache.spark.util.DatabricksUtils$.createSiteCustomizeScriptForVirtualenv(DatabricksUtils.scala:667)

at org.apache.spark.util.DatabricksUtils$.$anonfun$getOrCreatePythonVirtualEnvDir$1(DatabricksUtils.scala:723)

at org.apache.spark.util.DatabricksUtils$.withStripedLock(DatabricksUtils.scala:411)

at org.apache.spark.util.DatabricksUtils$.getOrCreatePythonVirtualEnvDir(DatabricksUtils.scala:708)

at com.databricks.backend.daemon.driver.DBUtilsPythonEnvManager.getPythonExecutable(DBUtilsPythonEnvManager.scala:27)

at com.databricks.backend.daemon.driver.PythonDriverLocal.launchPython(PythonDriverLocal.scala:315)

at com.databricks.backend.daemon.driver.PythonDriverLocal.startPython0(PythonDriverLocal.scala:956)

at com.databricks.backend.daemon.driver.PythonDriverLocal.startPython(PythonDriverLocal.scala:1009)

at com.databricks.backend.daemon.driver.PythonDriverLocal.<init>(PythonDriverLocal.scala:1032)

at com.databricks.backend.daemon.driver.PythonDriverWrapper.instantiateDriver(DriverWrapper.scala:697)

at com.databricks.backend.daemon.driver.DriverWrapper.setupRepl(DriverWrapper.scala:335)

at com.databricks.backend.daemon.driver.DriverWrapper.run(DriverWrapper.scala:224)

at java.lang.Thread.run(Thread.java:750)

Caused by: java.lang.Throwable: error=2, No such file or directory

at java.lang.UNIXProcess.forkAndExec(Native Method)

at java.lang.UNIXProcess.<init>(UNIXProcess.java:247)

at java.lang.ProcessImpl.start(ProcessImpl.java:134)

at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)

NOTE : All Python and %sh commands are giving this error and scala commands are running fine and i am able to see spark-scala logs in Log Analytics.

Someone please let me know what is wrong with Python Virtual enviroment and how can i resolve this error and run python commands in Databricks notebook successfully?

Thanks & Regards,

Akanksha

[akanksha_gupta]

Hi Everyone, Looking for urgent solution to the above mentioned problem. Please help if anyone has clues about the above error and it's resolution.

[Anonymous]

@Akanksha Gupta​ :

The error message suggests that the Python executable file specified in the configuration of the Databricks cluster cannot be found or accessed. Specifically, it seems that the Python executable file at the path "/local_disk0/pythonVirtualEnvDirs/virtualEnv-5acc1ea9-d03f-4de3-b76b-203d42614000/bin/python" is missing or inaccessible.

To resolve this issue, you may try the following steps:

1. Check if the path to the Python executable file is correct and accessible. You may need to check the configuration settings of your Databricks cluster and ensure that the correct Python version is installed.
2. Ensure that the Python environment is set up correctly and that all the required libraries and dependencies are installed.
3. Detach and re-attach the notebook or restart the Databricks cluster to see if the issue is resolved.