Python library not installed when compute is resized
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-19-2024 01:34 AM
Hi,
I have a python notebook workflow that uses a job cluster. The cluster lost at least a node (due to Spot Instance Termination) and did an upsize. After that I got an error in my job "Module not found", but the python module was being used before the cluster lost the node. So I guess the library was not installed in the new node. This is the first time it happens in this workflow when a node restarts. The cluster I use has 4 workers.
Any idea what might be going wrong? Thanks !