%pip/%conda doesn't work with encrypted clusters starting DBR 9.x

Prabakar
Databricks Employee
Databricks Employee

While trying to use the magic command %pip/%conda with DBR 9.x or above it fails with the following error:

 

%pip install numpy

org.apache.spark.SparkException: %pip/%conda commands use unencrypted NFS and are disabled by default when SSL encryption is enabled. NFS can be safely used to install libraries that do not contain PHI or other sensitive data, such as open source packages. %pip/%conda commands or NFS should not be used to transmit PHI to Spark workers. To enable %pip/%conda commands, set spark.databricks.conda.ignoreSSL to true in Spark config in cluster settings and restart your cluster.

 

How to resolve this?