Hey EndreM, Thanks a lot for the extra info!

Since you're running this from a Databricks notebook and still getting the Python version mismatch earlier, and now seeing a MissingCredentialScopeException, it strongly points to a misconfiguration related to Unity Catalog permissions or the identity used to access the data. The error suggests that Unity Catalog is expecting a credential scope (essentially, permission context) that’s not available in your current session—this usually happens when your cluster or user identity doesn't have access to the underlying external location or the credential isn't properly scoped to your workspace or catalog.

To fix this, here are a few suggestions:

• Check External Location Permissions: Make sure the external location (like your S3 path) is registered and you have the correct privileges set using GRANT in Unity Catalog.

• Verify Credential Scope Setup: Go to the Unity Catalog configuration in your workspace and confirm that the external location is linked with a valid credential and a storage credential scope. You can do this via the UI or with CREATE/ALTER EXTERNAL LOCATION and CREATE/GRANT STORAGE CREDENTIAL.

• Cluster Access Mode: If you’re using a shared or assigned cluster, make sure it’s in Shared Access Mode—this is required to work with Unity Catalog, especially with fine-grained permissions.

• Library Compatibility: Double-check that your custom library was rebuilt with Python 3.10 (the default in DBR 15.4), and no old .pyc or mismatched wheel files are lingering.

Once Unity Catalog permissions and scopes are configured properly, and your cluster is using the correct access mode, this kind of credential error should go away. Let me know if you want help checking the config or writing any of the SQL commands—I’m happy to help you get this sorted!

Regards,

Brahma