Unable to setcheckpointdir in unitycatalog enabled workspace

[reshmir18]

I have a Unity catalog enabled workspace where I am trying to setCheckpointDir during runtime. The method looks to authenticate using fs.azure.account.key instead of storage credentials. I am using databricks access connector which has "Storage Blob Data Contributor" role on my storage account. The relevant containers are also added as external locations.

Error:

Py4JJavaError: An error occurred while calling o350.setCheckpointDir.
: Failure to initialize configuration for storage account: Invalid configuration value detected for fs.azure.account.keyInvalid configuration value detected for fs.azure.account.key

[Kaniz_Fatma]

Hi @reshmir18, It seems you’re encountering an error related to setting the checkpoint directory in your Unity catalog-enabled workspace.

Let’s break down the issue and explore potential solutions:

Unity Catalog and Data Governance:

• Unity Catalog is a fine-grained governance solution for data and AI on the Databricks Platform. It simplifies security and governance by providing a central place to administer and audit data access.
• Data governance in Azure Databricks involves several building blocks, including metastores and catalogs. Metastores manage data assets (tables, views, and volumes) and their associated permissions. Catalogs represent logical groupings of schemas, often aligned with organizational units or developm....

Authentication and Access Modes:

• Unity Catalog introduces the concept of a cluster’s access mode. By default, Unity Catalog is secure. If a cluster is not configured with an appropriate access mode, it won’t be able to access data in Unity Catalog.
• Ensure that your cluster is configured correctly with the appropriate access mode to interact with U....

Storage Credentials and External Locations:

• Each storage credential in the Unity Catalog is subject to access-control policies. These policies control which users and groups can access the credentials.
• If a user lacks access to a storage credential in Unity Catalog, requests fail, and Unity Catalog wo...2.

Troubleshooting the Error:

• The error message you’re encountering suggests an issue with the configuration value for fs.azure.account.key.
• Verify the following:
  • Storage Account Key: Ensure that you’ve correctly set the storage account key in your configuration.
  • Storage Credentials: Confirm that the storage credentials are properly configured in the Unity Catalog.
  • Access Permissions: Check if the Databricks access connector has the necessary permissions (such as “Storage Blob Data Contributor”) on your storage account.
  • External Locations: Make sure the relevant containers are correctly added as external locations in the Unity Catalog.
  • Cluster Configuration: Review your cluster’s access mode settings to ensure it aligns with Unity Catalog requirements.

Next Steps:

• Double-check your configuration settings, paying attention to the storage account key and access permissions.
• If the issue persists, consider reaching out to Databricks support for further assistance.

Remember that Unity Catalog provides a powerful framework for data governance, but correct configuration and access management are crucial for seamless operation. Good luck troubleshooting, and may your data flow smoothly! 🚀

[reshmir18]

@Kaniz_Fatma I have provided all the necessary permissions and were able to browse through the folders of the container added as an external location.

I don't understand why the method setcheckpointdir looks for account key when the access is already provided using storage credentials for the storage account.