Quick question before diving in: are you running this through Spark Connect? The binary name spark-4.1.1-bin-hadoop3-connect suggests yes, and the stack trace confirms it. The NullPointerException on DelegatingCatalogExtension.name() because “this.delegate” is null points to the Iceberg catalog plugin not initializing on the server side.
Which version of delta-iceberg jar are you including? This matters because iceberg-spark-runtime has to be compiled against a specific Spark version and this exact pattern has happened before. Delta 2.4.0 docs note that “delta-iceberg is currently not available… since iceberg-spark-runtime does not support Spark 3.4 yet”, and the Delta 4.0.1 release notes explicitly state that “hudi and iceberg are currently not compatible with Spark 4.1, as support depends on upcoming releases.” (github.com/delta-io/delta/releases)
Does downgrading to Spark 3.5 with Delta 3.3.x change anything?