Databricks apps (streamlit) not able to install python libs

[ewe]

So, I have a databricks streamlit app that is not able to install any python lib defined on the requirements.txt.

Issue is not specific to one lib, tried other ones but no python lib can be installed. Anyone with similar issue to help ?

[2025-02-19 10:35:21] [Databricks] [ERROR] [] Error deploying app: error installing requirements

[2025-02-19 10:35:21] [Databricks] [INFO] Deployment ended in 9.778553176s

ERROR: No matching distribution found for great_expectations==0.18.22

ERROR: Could not find a version that satisfies the requirement great_expectations==0.18.22 (from versions: none)

▼WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/great-expectations/

[Brahmareddy]

Hi ewe,

How are you doing today?, As per understanding, It looks like your Databricks Streamlit app is unable to install Python libraries from requirements.txt, which could be due to network issues, package compatibility, or Databricks environment settings. First, check if your cluster has internet access by running !ping pypi.org in a notebook—if it fails, you might need to configure a proxy or VPC settings. Also, ensure the Databricks runtime supports the package versions you’re using; try installing great_expectations==0.18.22 manually with pip install in a Databricks notebook to see if it works. If installation fails due to a temporary name resolution error, restarting the cluster or using a different PyPI mirror (e.g., --index-url https://pypi.org/simple) might help. Lastly, if you’re on a Unity Catalog-enabled workspace, check if the Python environment settings allow installing external libraries. Hope this helps!

Regrads,

Brahma

[ewe]

Hi @Brahmareddy , thank you for the tips!

So, I tried the steps you mentioned, and funny thing is that on the notebooks, everything works. I can install libs and do external conenctions just fine. One thing I noticed from the databricks apps, is that the problem is not just on the list installation, but with any external connection. I created an sample app using no libs, on the code, I tried to do a get to google.com but its also not working, so I assume the tool has no external (internet) connections at all. 

I also tried using an ip with a pipy repo, and it says network is not reachable, so its not just a dns problem. problem is, I have no ideia on which infra these apps run. I assume its a managed kubernets cluster, but could not find the vm/container where the app runs so I could debug the network. If you have some idea on how I can find where its running, it will be helpfull.

Thanks!

[ewe · Accepted solution]

Hi, returning to inform how the issue was solved. A bit of context, looks like the databricks apps runs on the databricks/microsoft backend, and not on our clusters. The control of the network ingress rules is done via SEG (https://www.databricks.com/blog/announcing-egress-control-serverless-and-model-serving-workloads). To fix the issue, on the SEG for your workspace you need to add pipy endpoints. It worked for me when configured as follows: