I have a workflow that is running upon a job cluster and contains a task that requires prophet library from PyPI:

{
                "task_key": "my_task",
                "depends_on": [
                    {
                        "task_key": "<...>"
                    }
                ],
                "notebook_task": {
                    "notebook_path": "<...>",
                    "source": "WORKSPACE"
                },
                "job_cluster_key": "job_cluster",
                "libraries": [
                    {
                        "pypi": {
                            "package": "prophet==1.1.2"
                        }
                    }
                ],
                "timeout_seconds": 0,
                "email_notifications": {}
            },

Sometimes it works fine but sometimes I got the error below:

Run result unavailable: job failed with error message
 Library installation failed for library due to user error for pypi {
  package: "prophet==1.1.2"
}
. Error messages:
Library installation attempted on the driver node of cluster <...> and failed. Please refer to the following error message to fix the library or contact Databricks support. Error Code: DRIVER_LIBRARY_INSTALLATION_FAILURE. Error Message: org.apache.spark.SparkException: Process List(/databricks/python/bin/pip, install, prophet==1.1.2, --disable-pip-version-check) exited with code 1. ERROR: Could not install packages due to an OSError: [Errno 2] No such file or directory: '/databricks/python3/bin/f2py'

I saw suggestions to install in advance this library on a cluster. But I start my workflow in a job cluster (not an all-purpose cluster) so there is no ability to install something in advance. Weird thing is that sometimes it's ok and sometimes not.

So if there is a way to install library with a 100% guarantee on a shared job cluster it would be great!