Unity Catalog Volume as spark checkpoint location (in 2026)

[DB1To3]

Hi folks,

I'd like my checkpoint folder to be easily accessible to me in my UC volume files.  This was possible in Fabric OneLake (Files).  For whatever reason, it is not easy in databricks volumes.  I keep getting meaningless errors:

spark.sparkContext.setCheckpointDir(f"/Volumes/my_poc_catalog/my_poc_schema/my_cool_file_volume/Raw/Temp/Today")

Py4JJavaError: An error occurred while calling o437.setCheckpointDir. : java.io.IOException: Operation not permitted at java.io.UnixFileSystem.canonicalize0(Native Method) at java.io.UnixFileSystem.canonicalize(UnixFileSystem.java:177) at java.io.File.getCanonicalPath(File.java:626) at java.io.File.getCanonicalFile(File.java:651)

I've also tried the dbfs:/ schema and have no luck on that side either.

This seems like it should be easy to do in databricks, considering it was easy in OneLake (Files is their equivalent to Volumes in databricks).  My end goal is just to make sure that my "wip" files that are generated during "checkpoint" operations are prominently visible in the databricks saas Portal and I don't have to go hunting for them at some obscure abfss address in azure.

I've granted access, the spark compute runs as me, and I can write other output to that location, so no idea why the checkpoint is complainint.

There are other prior questions here in the community about this; but nobody seems to be offering authoritative advice. Why can't I store checkpoints in Volumes?  Any help would be appreciated.

[ThomazNeto · Accepted solution]

Hi,

Good news, this one has a definitive answer in the docs as of this year. The reason you keep hitting java.io.UnixFileSystem.canonicalize is that sc.setCheckpointDir goes through the JVM's local file APIs, and those don't understand the /Volumes FUSE path. The docs say that plainly for java.io.File and friends: they "do not support reading from or writing to Unity Catalog volumes or workspace files using standard file paths".
https://docs.databricks.com/aws/en/files/

What works, from the page "DataFrame checkpoints in volumes":

• Databricks Runtime 18.1 or above.
• Dedicated or standard access mode. "DataFrame checkpoints in volumes are not supported on serverless compute."
• On dedicated compute you can use sc.setCheckpointDir. On standard access mode, don't touch the SparkContext, set the Spark conf instead:

spark.conf.set("spark.checkpoint.dir", "/Volumes/my_poc_catalog/my_poc_schema/my_cool_file_volume/Raw/Temp/Today")
checkpointed_df = df.checkpoint()

https://docs.databricks.com/aws/en/volumes/volume-files#dataframe-checkpoints

So check two things: the runtime version (anything below 18.1 will fail the way you're seeing, whatever path scheme you try) and whether you're on standard access mode, where the conf route is the supported one. On serverless, df.checkpoint() isn't available at all, it's listed with the cache APIs as unsupported.
https://docs.databricks.com/aws/en/compute/serverless/limitations

Two side notes. If what you actually want is streaming, that's a different mechanism: .option("checkpointLocation", "/Volumes/...") has worked on volumes for a while and is the example the docs use. And "You can't access volumes from RDDs", so anything RDD-based will keep failing regardless.
https://docs.databricks.com/aws/en/structured-streaming/checkpoints
https://docs.databricks.com/aws/en/volumes/

Once it's on 18.1 with the conf set, the checkpoint files show up in Catalog Explorer under the volume exactly like you wanted.

[DB1To3]

Thanks for the thorough reply, and the link to "DataFrame checkpoints in volumes".  I hadn't found it.  We will try to upgrade to 18.1 as soon as possible.

Do you happen to have any suggestions for Spark 3.5 DBX 14.3 in the meantime?  
I tried setting checkpoints to the abfss:// scheme location (that is used internally) but UC doesn't seem to want me going around behind its back like that.

I guess I should also consider "external volumes", and that would expose the data to the saas portal as well. If I can't upgrade to 18.1 right away, I think this will be the plan B.

[ThomazNeto]

Hi,

Glad it helped. On 14.3, a few things to keep straight before you pick plan B.

The error you got isn't about managed vs external. sc.setCheckpointDir hands the path to the JVM's local file API (that java.io.File canonicalize in the stack), and the docs say those APIs don't work with /Volumes paths at all. So an external volume accessed through /Volumes/... will fail exactly the same way on 14.3. What changes with an external volume is that its files live in a location you also govern as a UC external location, and that's the door: point the checkpoint at the cloud URI, not the volume path.
https://docs.databricks.com/aws/en/files/

For that to work, two conditions from the docs. The abfss path has to be covered by an external location you have READ FILES and WRITE FILES on, otherwise UC blocks it, which is probably the "going behind its back" you hit. And the compute matters: on standard access mode "RDD APIs are not supported" and "You can't access volumes from RDDs", and RDD-style checkpointing is exactly what setCheckpointDir feeds. Use dedicated (single user) access mode for this until 18.1. On standard mode you'll keep fighting it.
https://docs.databricks.com/aws/en/compute/standard-limitations
https://docs.databricks.com/aws/en/volumes/

So plan B, concretely: external location on the container, external volume on a folder inside it, dedicated cluster on 14.3, sc.setCheckpointDir("abfss://.../that-folder/checkpoints"). The files then show up in Catalog Explorer under the volume, which was your original goal.

Let us know how 18.1 goes.

[DB1To3]

Right, I was going to use the abfss.  Databricks can't prevent me from accessing that, even if it is set up as an external volume. 

I can also point my on-prem Spark clusters at it, for that matter.

Thanks again.