I'm trying to switch checkpoint locations from dbfs to abfss and i have noticed the following behaviour.
The spark.sparkContext.setCheckpointDir will fail unless I call...
dbutils.fs.mkdirs(checkpoint_dir) in the same cell.
On top of this, the df = df.checkpoint(True) will fail, unless I run dbutils.fs.mkdirs(checkpoint_dir) in the same cell.
In both cases the error message is
Invalid configuration value detected for fs.azure.account.keyInvalid configuration value detected for fs.azure.account.key
if i include all three of the above in one cell and in the following order, it works
dbutils.fs.mkdirs(checkpoint_dir)
spark.sparkContext.setCheckpointDir(checkpoint_dir)
df = spark.table("cat.sch.tbl").checkpoint(True)
This is different behavior as compared to dbfs checkpoint locations
any one have any ideas?
