This is a classpath mismatch between the interactive cluster and the Workflow job cluster.

What I believe happened: the notebook was running on an all-purpose cluster with the Maven libraries attached, but the job was using a separate job cluster that did not have (or had conflicting versions of) spark-xml.

Fix

• Add the Maven libraries directly to the Workflow job cluster (or job YAML), not just the interactive cluster:

  • com.databricks:spark-xml_2.12:0.18.0

  • com.crealytics:spark-excel_2.12:3.4.3_0.20.4

• Ensure there are no duplicate or conflicting spark-xml jars

• Confirm the same DBR/Spark version is used for both notebook and job