Tree-based estimators in pyspark.ml have an argument called checkpointInterval

checkpointInterval

 = Param(parent='undefined', name='checkpointInterval', doc='set checkpoint interval (>= 1) or disable checkpoint (-1). E.g. 10 means that the cache will get checkpointed every 10 iterations. Note: this setting will be ignored if the checkpoint directory is not set in the SparkContext.')

When I run sc.getCheckpointDir() I get None so I'm assuming that the setting will be ignored as stated in the doc. Would it be a wrong assumption to make?

I've tried to set one with sc.setCheckpointDir("dbfs:/path/to/checkpoint"). After fitting an estimator I checked if there were any files in it with dbutils.fs.ls(sc.getCheckpointDir()) and nothing was there.

The doc of pyspark.SparkContext.setCheckpointDir says that "The directory must be an HDFS path if running on a cluster." But am I right that a DBFS paths should work too?

Is there a way to check if the estimator is indeed checkpointing at fitting time?