- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-18-2024 06:15 PM - edited 01-18-2024 06:16 PM
Volumes mount are accessible using scala code only on a shared cluster. On single user mode this features is not supported yet. We use init scripts to move contents from Volumes to clusters local drive, when we need to access files from Native Scala code. Volumes work out of the box for Spark APIs though - reading files, checkpoints, etc..
https://docs.databricks.com/en/files/index.html
Important
File operations that require FUSE access to data cannot directly access cloud object storage using URIs. Databricks recommends using Unity Catalog volumes to configure access to these locations for FUSE.
Scala does not support FUSE for Unity Catalog volumes or workspace files on compute configured with assigned access mode or clusters without Unity Catalog. Scala supports FUSE for Unity Catalog volumes and workspace files on compute configured with Unity Catalog and shared access mode.
Try below code on a Shared vs Single user cluster,
%scala
import java.nio.file.Paths
import scala.io.Source
import java.nio.file.{FileSystems, Files}
import scala.collection.JavaConverters._
Files.list(Paths.get("/Volumes")).iterator().asScala.foreach(println)