We face the same issue. Our code depends heavily on stored procedures (best practice, no).

Unfortunately, when moving to shared clusters and DBR 13.3, this does not work anymore.

driver_manager = spark._sc._gateway.jvm.java.sql.DriverManager
connection = driver_manager.getConnection(_MetaDBJDBCconnectionUrl, _MetaDBUserName, _MetaDBPassword)
connection.prepareCall(sqlExecuteStatement).execute()
connection.close()

now throws the error

An error occurred while calling z:java.sql.DriverManager.getConnection. Trace:
py4j.security.Py4JSecurityException: Method public static java.sql.Connection java.sql.DriverManager.getConnection(java.lang.String,java.lang.String,java.lang.String) throws java.sql.SQLException is not whitelisted on class class java.sql.DriverManager

as listed above.

Going for approach 3 does not work in our case.

Any other suggestions?