Walter_C
Databricks Employee
Databricks Employee

Here are a few possible explanations and solutions:

  1. Transient Issue: Considering that the error was a one-off and the job has been running fine since then, it's possible that it was a transient issue. Transient issues can occur due to temporary network glitches, issues with the PyPi server at the time of the job run, or other temporary problems.

  2. Cluster Initialization Timing: Sometimes, if a job starts running before all the libraries have been fully installed on the cluster, it can lead to a ModuleNotFoundError. This is more likely to happen if the cluster is just starting up and the job starts running immediately.

  3. Package Installation Failure: There might been an issue with the installation of the package for that particular run. You can check the cluster logs for any errors or warnings related to the package installation.

  4. Package Compatibility Issue: Ensure that the package is compatible with the Python version and the Databricks runtime version you're using.