dataframe checkpoint when checkpoint location on abfss

Leigh_Turner
New Contributor

‎08-08-2024 06:48 AM

 

I'm trying to switch checkpoint locations from dbfs to abfss and i have noticed the following behaviour.

The spark.sparkContext.setCheckpointDir will fail unless I call...

dbutils.fs.mkdirs(checkpoint_dir) in the same cell.

On top of this, the df = df.checkpoint(True) will fail, unless I run dbutils.fs.mkdirs(checkpoint_dir) in the same cell.
In both cases the error message is

Invalid configuration value detected for fs.azure.account.keyInvalid configuration value detected for fs.azure.account.key

if i include all three of the above in one cell and in the following order, it works

dbutils.fs.mkdirs(checkpoint_dir)
spark.sparkContext.setCheckpointDir(checkpoint_dir)
df = spark.table("cat.sch.tbl").checkpoint(True)

This is different behavior as compared to dbfs checkpoint locations

any one have any ideas?

0 Kudos

Mounika_Tarigop
Databricks Employee
Databricks Employee

‎12-06-2024 10:31 PM

In DBFS, the checkpoint directory is automatically created when you set it using spark.sparkContext.setCheckpointDir(checkpoint_dir). This means that you do not need to explicitly create the directory beforehand using dbutils.fs.mkdirs(checkpoint_dir).

However, when using ABFSS, the directory does not get created automatically. You need to explicitly create the checkpoint directory using dbutils.fs.mkdirs(checkpoint_dir) before setting it with spark.sparkContext.setCheckpointDir(checkpoint_dir). This ensures that the directory exists and is accessible, which is why including dbutils.fs.mkdirs(checkpoint_dir) in the same cell is necessary for ABFSS but not for DBFS.

0 Kudos