Models signature inaccessible in Unity Catalog

[owlleg6]

I am using Unity Catalog to store my models. My Unity Catalog meta-store is hosted in an Azure Storage Account, which has public network access disabled. Access is restricted to certain IP ranges and private endpoint connections only.

Recently, my development team has noticed that the signatures of the models are inaccessible, although we can access every table, volume, etc., using the Private Endpoint connected to the Storage Account. Interestingly, if I open the Storage Account to the public network, the model signatures become accessible. Could you please explain why all folders within the Unity Catalog meta-store are accessible except for the 'models' folder?

[owlleg6]

This exception from mlflow when downloading artifact

[Kaniz_Fatma]

Hi @owlleg6 , Thank you for sharing your scenario regarding Unity Catalog, Azure Storage Account, and private endpoints. 

Let’s explore the intricacies of this situation:

Unity Catalog and Azure Storage:

• Unity Catalog serves as an essential component in Databricks data governance. It manages data assets such as tables, views, and volumes, along with permissions governing access.
• Your Unity Catalog meta-store resides in an Azure Storage Account. This storage account has public network access disabled, ensuring security by restricting access to specific IP ranges and private endpoints.

Private Endpoints and Model Signatures:

• Private endpoints provide additional security and control over network traffic to and from your storage account.
• When you create a private endpoint for your storage account, all traffic to and from the storage account is routed through this endpoint. Clients and applications must use the private endpoint to connect.
• Now, here’s the interesting part: While other folders within the Unity Catalog meta-store are accessible via the private endpoint, the ‘models’ folder remains elusive.

Potential Reasons for Inaccessibility:

• The specific reason for the inaccessibility of the ‘models’ folder might involve how the Unity Catalog interacts with the storage account.
• It’s possible that the models’ folder contains sensitive information or requires a different level of access control.
• Consider checking the permissions, ACLs, and security settings specifically for the ‘models’ folder. There might be unique configurations affecting its accessibility.

Performance and Latency Considerations:

• While private endpoints enhance security, they can introduce additional latency or network overhead. The impact depends on your network configuration and client locations.
• Evaluate the trade-offs between security and performance. The private endpoint ensures data exfiltration prevention but may affect response times.

Testing and Configuration:

• To troubleshoot, consider:
  • Reviewing permissions: Ensure that the necessary permissions are granted for the ‘models’ folder.
  • Testing: Temporarily allow public network access to verify if the issue persists.
  • Azure-managed identities: Explore using Azure-managed identities for Unity Catalog to access storage....
  • Thorough testing: Validate the new configuration to meet your requirements.

If you need further assistance, feel free to ask!

[owlleg6]

Hello, your moderators are deleting my previous replies, that's funny something is not working properly, i am trying to find out the solution but you are deleting the posts.
So, the issue is with a networking for sure, not with the permissions. Where can i raise a ticket? This is critical issue for my team, we have to disable public access to Storage Account.

Cheers

[Rikard007]

We are facing the same issue. We can’t reconsider security setup since that is enforced by IT security. Does anyone have suggestions on how to resolve this issue so we can save and access the signatures from models stored in UC? 

[lbdatauser]

I have the same problem. It must be a permissions issue. In my case, it happens when the models are under a schema that was created without indicating a specific "MANAGED LOCATION". In those cases, I also have problems trying to load a model. When the models are under a schema created with a particular "MANAGED LOCATION" all works ok for me.

[owlleg6]

Fair point, what if i don't want to use External Locations for Schemas where models will be stored? Why default Metastore location does not allow accessing signatures.