DAB wheel installation job fails, user error Library from /Workspace not allowed

[AndyM]

Hi Community!

I am getting started with DABs and just recently ran into a following error after deployment trying to run my bundle that has a wheel installation job. 

Error: failed to reach TERMINATED or SKIPPED, got INTERNAL_ERROR: Task main_task failed with message: Library installation failed for library due to user error. Error messages:
Library from /Workspace is not allowed on non Unity Catalog cluster.
Please switch to DBR 13.1+ Shared cluster or 13.2+ Assigned cluster to use /Workspace libraries.

Is it so that it is not possible to deploy into a workspace folder and use that to install a wheel if unity catalog is not configured? If possible how could I switch my DBR (13.3) to use /Workspace libraries? From DABs documentation I was not able to find information whether UC is required.

[Kaniz]

Hi @AndyM , 

The error message you encountered indicates that the library installation failed due to a restriction related to the Unity Catalog. Let’s break down the issue and explore the solution:

1. Unity Catalog and Workspace:

   • Unity Catalog is a data governance feature in Databricks that helps manage data assets.
   • To use Unity Catalog, your Databricks workspace must be enabled for Unity Catalog. This means that the workspace is attached to a Unity Catalog metastore, which serves as the top-level container for Unity Catalog metadata.
   • The Unity Catalog metastore is configured with a managed storage location (usually an S3 bucket in your AWS account).

2. Automatic vs. Manual Enablement:

   • Workspaces created after November 8, 2023 are automatically enabled for Unity Catalog. These workspaces have certain default properties.
   • Manually-enabled workspaces require specific steps to enable Unity Catalog.

3. Your Scenario:

   • You encountered the error because your cluster is not configured to use Unity Catalog.
   • The library you’re trying to install is located in the /Workspace folder, which is not allowed on non-Unity Catalog clusters.

4. Solution:

   • To resolve this, follow these steps:
     • Confirm Workspace Enablement: Make sure your workspace is enabled for Unity Catalog. If it’s not, you’ll need to attach it to a Unity Catalog metastore.
     • Create Unity Catalog Clusters: Create clusters or SQL warehouses that have access to Unity Catalog. These clusters will allow you to run queries and create objects.
     • Grant Privileges: Ensure that users have the necessary permissions to access and create objects in Unity Catalog.
     • Create Catalogs and Schemas: Optionally, create new catalogs and schemas within Unity Catalog.
     • Assign Metastore Admin Role: If needed, assign the metastore admin role.

5. DBR Version and Unity Catalog:

   • Unity Catalog is supported on clusters running Databricks Runtime 11.3 LTS or above.
   • Clusters running on earlier versions of Databricks Runtime may not provide full support for Unity Catalog features.

6. Additional Resources:

   • For more details, refer to the official Databricks documentation on Unity Catalog.
   • If you encounter any issues during setup, please LMK.

Remember that Unity Catalog enhances data governance, so configuring it correctly ensures a smoother experience when managing your data assets. 🌟