Hi, my issue was "resolved" when i switched to personal compute with 14.3 LTS ML Runtime. OAuth did not seem to make a difference since i still needed to use drivermanager at the end of the day. So if you need a temporary solution i would suggest trying that, but i do hope that there are plans for fixing this in the future.
# Write your SQL statement as a string
statement = """TRUNCATE TABLE dbo.test"""
# Fetch the driver manager from your spark context
driver_manager = spark._sc._gateway.jvm.java.sql.DriverManager
# Create a connection object using a jdbc-url, + sql uname & pass
con = driver_manager.getConnection(jdbc_url, user, password)
# Create callable statement and execute it
exec_statement = con.prepareCall(statement)
exec_statement.execute()
# Close connections
exec_statement.close()
con.close()