cancel
Showing results for 
Search instead for 
Did you mean: 

DAB wheel installation job fails, user error Library from /Workspace not allowed

AndyM
New Contributor II

Hi Community!

I am getting started with DABs and just recently ran into a following error after deployment trying to run my bundle that has a wheel installation job. 

Error: failed to reach TERMINATED or SKIPPED, got INTERNAL_ERROR: Task main_task failed with message: Library installation failed for library due to user error. Error messages:
Library from /Workspace is not allowed on non Unity Catalog cluster.
Please switch to DBR 13.1+ Shared cluster or 13.2+ Assigned cluster to use /Workspace libraries.

Is it so that it is not possible to deploy into a workspace folder and use that to install a wheel if unity catalog is not configured? If possible how could I switch my DBR (13.3) to use /Workspace libraries? From DABs documentation I was not able to find information whether UC is required.

1 REPLY 1

Kaniz
Community Manager
Community Manager

Hi @AndyM , 

The error message you encountered indicates that the library installation failed due to a restriction related to the Unity Catalog. Let’s break down the issue and explore the solution:

  1. Unity Catalog and Workspace:

    • Unity Catalog is a data governance feature in Databricks that helps manage data assets.
    • To use Unity Catalog, your Databricks workspace must be enabled for Unity Catalog. This means that the workspace is attached to a Unity Catalog metastore, which serves as the top-level container for Unity Catalog metadata.
    • The Unity Catalog metastore is configured with a managed storage location (usually an S3 bucket in your AWS account).
  2. Automatic vs. Manual Enablement:

    • Workspaces created after November 8, 2023 are automatically enabled for Unity Catalog. These workspaces have certain default properties.
    • Manually-enabled workspaces require specific steps to enable Unity Catalog.
  3. Your Scenario:

    • You encountered the error because your cluster is not configured to use Unity Catalog.
    • The library you’re trying to install is located in the /Workspace folder, which is not allowed on non-Unity Catalog clusters.
  4. Solution:

    • To resolve this, follow these steps:
      • Confirm Workspace Enablement: Make sure your workspace is enabled for Unity Catalog. If it’s not, you’ll need to attach it to a Unity Catalog metastore.
      • Create Unity Catalog Clusters: Create clusters or SQL warehouses that have access to Unity Catalog. These clusters will allow you to run queries and create objects.
      • Grant Privileges: Ensure that users have the necessary permissions to access and create objects in Unity Catalog.
      • Create Catalogs and Schemas: Optionally, create new catalogs and schemas within Unity Catalog.
      • Assign Metastore Admin Role: If needed, assign the metastore admin role.
  5. DBR Version and Unity Catalog:

    • Unity Catalog is supported on clusters running Databricks Runtime 11.3 LTS or above.
    • Clusters running on earlier versions of Databricks Runtime may not provide full support for Unity Catalog features.
  6. Additional Resources:

    • For more details, refer to the official Databricks documentation on Unity Catalog.
    • If you encounter any issues during setup, please LMK.

Remember that Unity Catalog enhances data governance, so configuring it correctly ensures a smoother experience when managing your data assets. 🌟

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.