How to configure a Job-Compute for Unity Catalog Access? (Q/A)

[PabloCSD]

If you need to access tables that are in a volume of Unity Catalog (UC), with the following configuration will work:

targets:
  dev:
    mode: development
    default: true
    workspace:
      host: https://<workspace>.azuredatabricks.net/
    run_as:
      user_name: <user_email>
    resources:
      jobs:
        etl_process:
          job_clusters:
            - job_cluster_key: ${bundle.target}-${bundle.name}-job-cluster
              new_cluster:
                autoscale:
                  min_workers: 2
                  max_workers: 8
                spark_version: "14.3.x-scala2.12"
                driver_node_type_id: "Standard_F16s_v2"
                node_type_id: "Standard_L4s"
                data_security_mode: "SINGLE_USER"
                runtime_engine: "STANDARD"
                azure_attributes:
                  first_on_demand: 1
                  availability: SPOT_WITH_FALLBACK_AZURE
                  spot_bid_max_price: 100
                spark_conf:
                  spark.databricks.unityCatalog.enabled: "true"
                  spark.sql.catalog.unity: "com.databricks.sql.managedcatalog.UnityCatalogProvider"
                  spark.sql.catalog.spark_catalog: "org.apache.spark.sql.delta.catalog.DeltaCatalog"

[Khaja_Zaffer]

Hello @PabloCSD 

Good day!

Are you asking or like what are you expectations?

Additions to this: 

You cannot create or register tables (managed or external) with locations pointing to volumes, as this is explicitly not supported—tables must use tabular storage locations instead.

To access data within UC volumes, use path-based methods (e.g., spark.read.load('/Volumes/catalog/schema/volume/path/to/file') or dbutils.fs.ls('/Volumes/...')), assuming the user specified in run_as has the required UC privileges like READ VOLUME or WRITE VOLUME on the volume.