Databricks Community

Miguel_Salas · a month ago

I'm trying to install Pyrfc in a Databricks Cluster (already tried in r5.xlarge, m5.xlarge, and c6gd.xlarge). I'm following these link.

https://community.databricks.com/t5/data-engineering/how-can-i-cluster-install-a-c-python-library-py...

But I am still having problems installing Pyrfc

Set environment variables in cluster sets, and put an init script in cluster.

Miguel_Salas · a month ago

More details about the error

Library installation attempted on the driver node of cluster 0000-000000-00000 and failed. Please refer to the following error message to fix the library or contact Databricks support. Error code: DRIVER_LIBRARY_INSTALLATION_FAILURE. Error message: org.apache.spark.SparkException: Process List(/bin/su, libraries, -c, bash /local_disk0/.ephemeral_nfs/cluster_libraries/python/python_start_clusterwide.sh /local_disk0/.ephemeral_nfs/cluster_libraries/python/bin/pip install 'pyrfc==3.3.1' --disable-pip-version-check) exited with code 1.   Running command pip subprocess to install build dependencies
  Using pip 23.2.1 from /local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.11/site-packages/pip (python 3.11)
  Non-user install by explicit request
  Created build tracker: /tmp/pip-build-tracker-xk61ox_k
  Entered build tracker: /tmp/pip-build-tracker-xk61ox_k
  Created temporary directory: /tmp/pip-install-y24e5y8f
  Created temporary directory: /tmp/pip-ephem-wheel-cache-462pfql0
  Looking in indexes: https://pypi.org/simple,