VZLA
Databricks Employee
Databricks Employee

To connect to Databricks Unity Catalog from a local Spark session, you'll need to configure your Spark session with the appropriate dependencies, extensions, and authentication. Here's a general setup:

  1. Install the required libraries, such as io.delta:delta-spark_2.12:3.2.1 and io.unitycatalog:unitycatalog-spark_2.12:0.2.0. You'll need to check for the right compatible versions, try looking into your cluster's /databricks/jars or Drivers Classpath for matching library versions.
  2. Configure your Spark session using the following settings:
  3. --name "local-uc-test"
    --master "local[*]"
    --conf "spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension"
    --conf "spark.sql.catalog.spark_catalog=io.unitycatalog.spark.UCSingleCatalog"
    --conf "spark.sql.catalog.unity.uri=<YOUR_UNITY_CATALOG_ENDPOINT>"
    --conf "spark.sql.catalog.unity.token=<YOUR_ACCESS_TOKEN>"
    --conf "spark.sql.defaultCatalog=unity"
  4. For authentication, use a valid personal access token for spark.sql.catalog.unity.token. You can generate one in your Databricks workspace with appropriate permissions (Settings -> Developer -> Generate New Token).
  5. Once the Spark session starts, validate the setup by running SHOW CATALOGS in a Spark SQL query to confirm access to Unity Catalog.

If any issues arise, ensure network connectivity to the Unity Catalog endpoint and verify that your environment has the necessary configurations.