Unity Catalog Volume as spark checkpoint location

[Balazs]

Hi,

I tried to set the spark checkpoint location in a notebook to a folder in a Unity Catalog Volume, with the following command: 

sc.setCheckpointDir("/Volumes/catalog_name/schema_name/volume_name/folder_name")

Unfortunately I receive the following error: "Py4JJavaError: An error occurred while calling o356.setCheckpointDir. : java.io.IOException: Operation not permitted".

My user have all privileges granted on the volume.

Did anyone face the same issue? Is it possible to use Databricks volumes as storage location for checkpoints?

[Kaniz]

Hi @Balazs , Databricks volumes are Unity Catalog objects that represent logical volumes of storage in a cloud object storage location. They provide capabilities for accessing, storing, governing, and organizing files. While tables govern tabular datasets, volumes add governance over non-tabular datasets, allowing you....

Regarding your specific issue with setting the checkpoint location, consider checking the following:

• Permissions on the specified folder.
• Whether the folder exists in the Unity Catalog volume.
• Whether the user has the required privileges for checkpointing.

Remember that Databricks volumes provide a powerful way to organize and manage your data, and they can indeed be used as storage locations for checkpoints.

: [Databricks Documentation: Volumes](https://docs.databricks.com/en/sql/language-manual/sql-ref-volumes.html