I do not think it is loading Spark 3.2. I am still seeing the issue with writeUTF which has been fixed in Spark 3.2 -> https://github.com/apache/spark/pull/32788

Caused by: java.io.UTFDataFormatException: encoded string too long: 97548 bytes
	at java.io.DataOutputStream.writeUTF(DataOutputStream.java:364)
	at java.io.DataOutputStream.writeUTF(DataOutputStream.java:323)

Anyways, I will wait for the databricks runtime to correctly reflect the correct version.