Global init script fails on Databricks 16.0

[k1t3k]

#!/bin/bash
pip install package1 --index-url https://link-to-index
pip install package2 --index-url https://link-to-index

This init script fails with

error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
   python3-xyz, where xyz is the package you are trying to
   install.

   If you wish to install a non-Debian-packaged Python package,
   create a virtual environment using python3 -m venv path/to/venv.
   Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
   sure you have python3-full installed.

   If you wish to install a non-Debian packaged Python application,
   it may be easiest to use pipx install xyz, which will manage a
   virtual environment for you. Make sure you have pipx installed.

   See /usr/share/doc/python3.12/README.venv for more information.

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.

[notice] A new release of pip is available: 24.0 -> 24.3.1
[notice] To update, run: python3.12 -m pip install --upgrade pip
error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
   python3-xyz, where xyz is the package you are trying to
   install.

   If you wish to install a non-Debian-packaged Python package,
   create a virtual environment using python3 -m venv path/to/venv.
   Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
   sure you have python3-full installed.

   If you wish to install a non-Debian packaged Python application,
   it may be easiest to use pipx install xyz, which will manage a
   virtual environment for you. Make sure you have pipx installed.

   See /usr/share/doc/python3.12/README.venv for more information.

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.

[notice] A new release of pip is available: 24.0 -> 24.3.1
[notice] To update, run: python3.12 -m pip install --upgrade pip

This global init script works fine with databricks runtime 15.4LTS, but crashes with 16.0

[Alberto_Umana]

Hi @k1t3k,

Are you installing a custom package? could you please share the package name you are installing to validate?

The error you are encountering, "externally-managed-environment," when running your global init script with Databricks Runtime 16.0, is likely due to changes in how Python environments are managed in this runtime version.

[k1t3k]

These are packages maintained by our company and I cannot share details on them.

So how can I install my package to an environment used by databricks runtime? python3 -m pip install succeeds, but it's definitely not the right environment since the package doesn't work when installed like this