To fix this issue, we can use the below init script to download the instant client and add that in the path variable. Sometimes the dynamic linker run-time cache needs to be refreshed, especially if there are native libraries that have cross dependencies, and they were placed on the machine before the LD_LIBRARY_PATH was set. In this case, run ldconfig -v /path/to/native/libs before attempting to load the native library from Java.

Here is a sample init script that would help

Step 1: Create the base directory you want to store the init script ( assuming it does not exist.) Here we use dbfs:/databricks/<directory> as an example.

              dbutils.fs.mkdirs("dbfs:/databricks/<directory>/")

Step 2 : Create the script 

 
 
dbutils.fs.put("dbfs:/databricks/oracleTest/oracle_ctl.sh","""
 
 
 
#!/bin/bash
 
 
 
#Download instant client archived file and update the url in case of different #version
 
 
 
wget --quiet-O /tmp/instantclient-basiclite-linux.x64-19.3.0.0.0dbru.zip https://download.oracle.com/otn_software/linux/instantclient/193000/instantclient-basiclite-linux.x6...
 
 
 
unzip /tmp/instantclient-basiclite-linux.x64-19.3.0.0.0dbru.zip -d /databricks/driver/oracle_ctl/
 
 
 
sudo echo 'export LD_LIBRARY_PATH="/databricks/driver/oracle_ctl/"' >> /databricks/spark/conf/spark-env.sh
 
 
 
sudo echo 'export ORACLE_HOME="/databricks/driver/oracle_ctl/"' >> /databricks/spark/conf/spark-env.sh
 
 
 
""", True)

Step 3: Verify that the script exists.

              display(dbutils.fs.ls("dbfs:/databricks/<directory>/oracle_ctl.sh"))

Step 4. Configure a cluster-scoped init script in the cluster 

• On the cluster configuration page, click the Advanced Options toggle.
• At the bottom of the page, click the Init Scripts tab.
• In the Destination drop-down, select DBFS, provide the file path to the script, and click Add.

Step 5: Restart the cluster