Unity Catalog Volume as spark checkpoint location
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-24-2023 06:05 AM
Hi,
I tried to set the spark checkpoint location in a notebook to a folder in a Unity Catalog Volume, with the following command:
sc.setCheckpointDir("/Volumes/catalog_name/schema_name/volume_name/folder_name")
Unfortunately I receive the following error: "Py4JJavaError: An error occurred while calling o356.setCheckpointDir. : java.io.IOException: Operation not permitted".
My user have all privileges granted on the volume.
Did anyone face the same issue? Is it possible to use Databricks volumes as storage location for checkpoints?