Your YAML is valid, but the reason the libraries are not being installed is because base_environment_path is not supported for serverless compute. Serverless jobs use a fully managed environment and you canโt override it with a custom base environment.
If you need specific libraries, you have two options:
Use a non-serverless cluster and point base_environment_path to your YAML.
For serverless, install the required packages either by defining them under libraries in the job spec or by using %pip install inside the notebook.
In short, base_environment_path has no effect on serverless jobs.
Yogesh Verma