Global init script fails on Databricks 16.0
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-14-2024 01:31 AM
#!/bin/bash
pip install package1 --index-url https://link-to-index
pip install package2 --index-url https://link-to-index
This init script fails with
error: externally-managed-environment
× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
python3-xyz, where xyz is the package you are trying to
install.
If you wish to install a non-Debian-packaged Python package,
create a virtual environment using python3 -m venv path/to/venv.
Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
sure you have python3-full installed.
If you wish to install a non-Debian packaged Python application,
it may be easiest to use pipx install xyz, which will manage a
virtual environment for you. Make sure you have pipx installed.
See /usr/share/doc/python3.12/README.venv for more information.
note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.
[notice] A new release of pip is available: 24.0 -> 24.3.1
[notice] To update, run: python3.12 -m pip install --upgrade pip
error: externally-managed-environment
× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
python3-xyz, where xyz is the package you are trying to
install.
If you wish to install a non-Debian-packaged Python package,
create a virtual environment using python3 -m venv path/to/venv.
Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
sure you have python3-full installed.
If you wish to install a non-Debian packaged Python application,
it may be easiest to use pipx install xyz, which will manage a
virtual environment for you. Make sure you have pipx installed.
See /usr/share/doc/python3.12/README.venv for more information.
note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.
[notice] A new release of pip is available: 24.0 -> 24.3.1
[notice] To update, run: python3.12 -m pip install --upgrade pip
This global init script works fine with databricks runtime 15.4LTS, but crashes with 16.0
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-14-2024 08:31 AM - edited 11-14-2024 08:33 AM
Hi @k1t3k,
Are you installing a custom package? could you please share the package name you are installing to validate?
The error you are encountering, "externally-managed-environment," when running your global init script with Databricks Runtime 16.0, is likely due to changes in how Python environments are managed in this runtime version.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-15-2024 12:23 AM
These are packages maintained by our company and I cannot share details on them.
So how can I install my package to an environment used by databricks runtime? python3 -m pip install succeeds, but it's definitely not the right environment since the package doesn't work when installed like this

