Not able to read data from Delta External table in catalog

bohemiaRDX — Wed, 05 Feb 2025 05:33:00 GMT

spark.conf.set(

"fs.azure.account.key.sa02flexflowinpp01prod.dfs.core.windows.net",

dbutils.secrets.get(scope="OpenScope", key="sa02StorageAccessKey")

I created an external table using this configuration I am able to query the data only when I run the above configuration. Also I am not able to see the sample data in catalog

)

Re: Not able to read data from Delta External table in catalog

Ayushi_Suthar — Wed, 05 Feb 2025 14:41:43 GMT

Hi @bohemiaRDX , Greetings!

Generally, this error would occur if the path is not added as an external location with storage credentials. Here the cluster could be trying to access the storage which doesn’t have UC storage credentials set nor any non UC access mechanisms like access keys/service principal setup and eventually failing with the given error.

What’s Next:

Verify if the given storage location needs to be accessed via unity catalog or not.
If the usage is for unity catalog,

ensure that storage credentials and external location are set for the path access. Refer to https://learn.microsoft.com/en-us/azure/databricks/storage/azure-storage#--connect-to-azure-data-lake-storage-gen2-with-unity-catalog and https://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/manage-external-locations-and-credentials for more information.
For table access failures, Verify if the schema has been set to UC one. This error may happen if it uses hive metastore catalog (may happen if there is no explicitl catalog and schema set + default catalog points to hive_metastore). Below query is useful to find the current catalog and schema used.

Select current_catalog(), current_schema()

If the usage is non UC based, ensure that relevant access configuration are in place. It can be set using OAuth 2.0 with an Azure service principal/SAS token/Account keys. These are usually set at notebook level/cluster spark configuration via UI/init script. For SQL warehouse, these are set in SQL warehouse settings for data access configuration.
Refer to https://learn.microsoft.com/en-us/azure/databricks/storage/azure-storage#connect-to-azure-data-lake-storage-gen2-or-blob-storage-using-azure-credentials for more details.

Leave a like if this helps, followups are appreciated.
Kudos,

Ayushi

topic Re: Not able to read data from Delta External table in catalog in Data Engineering

Not able to read data from Delta External table in catalog

Re: Not able to read data from Delta External table in catalog