Classloader shadowing in Databricks Runtime (DBR) is a common headache when the built-in connector JARs take precedence over cluster-installed libraries.Instead of manually stripping and rebuilding the JAR, try setting spark.driver.userClassPathFirst...
You aren't doing anything wrong logically, but Databricks requires row-level changes to happen on Delta Lake tables. Standard Spark tables (like those backed by raw Parquet) are often immutable. Have a look at the Language Manual for DELETE to ensure...