dataframe checkpoint when checkpoint location on abfss

Leigh_Turner — Thu, 08 Aug 2024 13:48:57 GMT

I'm trying to switch checkpoint locations from dbfs to abfss and i have noticed the following behaviour.

The spark.sparkContext.setCheckpointDir will fail unless I call...

dbutils.fs.mkdirs(checkpoint_dir) in the same cell.

On top of this, the df = df.checkpoint(True) will fail, unless I run dbutils.fs.mkdirs(checkpoint_dir) in the same cell.
In both cases the error message is

Invalid configuration value detected for fs.azure.account.keyInvalid configuration value detected for fs.azure.account.key

if i include all three of the above in one cell and in the following order, it works

dbutils.fs.mkdirs(checkpoint_dir)
spark.sparkContext.setCheckpointDir(checkpoint_dir)
df = spark.table("cat.sch.tbl").checkpoint(True)

This is different behavior as compared to dbfs checkpoint locations

any one have any ideas?

Re: dataframe checkpoint when checkpoint location on abfss

Mounika_Tarigop — Sat, 07 Dec 2024 06:31:51 GMT

In DBFS, the checkpoint directory is automatically created when you set it using spark.sparkContext.setCheckpointDir(checkpoint_dir). This means that you do not need to explicitly create the directory beforehand using dbutils.fs.mkdirs(checkpoint_dir).

However, when using ABFSS, the directory does not get created automatically. You need to explicitly create the checkpoint directory using dbutils.fs.mkdirs(checkpoint_dir) before setting it with spark.sparkContext.setCheckpointDir(checkpoint_dir). This ensures that the directory exists and is accessible, which is why including dbutils.fs.mkdirs(checkpoint_dir) in the same cell is necessary for ABFSS but not for DBFS.

topic Re: dataframe checkpoint when checkpoint location on abfss in Data Engineering

dataframe checkpoint when checkpoint location on abfss

Re: dataframe checkpoint when checkpoint location on abfss