Hi folks,
I'd like my checkpoint folder to be easily accessible to me in my UC volume files. This was possible in Fabric OneLake (Files). For whatever reason, it is not easy in databricks volumes. I keep getting meaningless errors:
spark.sparkContext.setCheckpointDir(f"/Volumes/my_poc_catalog/my_poc_schema/my_cool_file_volume/Raw/Temp/Today")
Py4JJavaError: An error occurred while calling o437.setCheckpointDir. : java.io.IOException: Operation not permitted at java.io.UnixFileSystem.canonicalize0(Native Method) at java.io.UnixFileSystem.canonicalize(UnixFileSystem.java:177) at java.io.File.getCanonicalPath(File.java:626) at java.io.File.getCanonicalFile(File.java:651)
I've also tried the dbfs:/ schema and have no luck on that side either.
This seems like it should be easy to do in databricks, considering it was easy in OneLake (Files is their equivalent to Volumes in databricks). My end goal is just to make sure that my "wip" files that are generated during "checkpoint" operations are prominently visible in the databricks saas Portal and I don't have to go hunting for them at some obscure abfss address in azure.
I've granted access, the spark compute runs as me, and I can write other output to that location, so no idea why the checkpoint is complainint.
There are other prior questions here in the community about this; but nobody seems to be offering authoritative advice. Why can't I store checkpoints in Volumes? Any help would be appreciated.
