Did you get any solution for the above issue? I am also trying same in DBR 15.4, Standard cluster .So I am able to set checkpoint directory using below commands.

spark.conf.set("pyspark.sql.DataFrame.checkpoint", "/Volumes/path/")
spark.conf.set("spark.sql.checkpoint.dir","/Volumes/path/")
spark.conf.set("spark.sql.checkpointLocation","/Volumes/path/")

 But its failing  while checkpointing a dataframe.

df.checkpoint(True)
-->Checkpoint directory has not been set in the SparkContext
File <command-8168308127814448>, line 1
----> 1 df.checkpoint(True)
File /databricks/spark/python/pyspark/sql/connect/client/core.py:2149, in SparkConnectClient._handle_rpc_error(self, rpc_error)
   2134                 raise Exception(
   2135                     "Python versions in the Spark Connect client and server are different. "
   2136                     "To execute user-defined functions, client and server should have the "

 In this case can we use localCheckoint as an alternative but I know localCheckoint are not reliable as  Local checkpoints are stored in the executors using the caching subsystem.

Is it not really possible to use checkpoints on UC enabled cluster with DBR 15.4 Or is there any new way to use checkpoint on dataframe .