topic Re: Delta sharing open protocol in Unity catalog: FileNotFoundError in Data Engineering

Delta sharing open protocol in Unity catalog: FileNotFoundError

kiko_roy — Tue, 11 Jun 2024 11:08:08 GMT

Hi Team

I have created a recipient under delta sharing (azure databricks) . Unity catalog is enabled and data is stored in ADLS gen2. I have downloaded the credential file and trying to resue in my python script (as per databricks documentation) for a POC activity from my local machine (jupyter notebook) . delta_sharing package is successfully installed in my local system

import delta_sharing
import pandas

client = delta_sharing.SharingClient(f"C:/Users/XXXXXX/Downloads/config.share")

client.list_all_tables()

delta_sharing.load_as_pandas(f"C:/Users/XXXXX/Downloads/config.share#<sharename>.<schemaname>.<tablename>")

I am able to see the successful listing of the tables, but when trying to load the a particular table data as a pandas dataframe , getting error :

FileNotFoundError: https://......................._unitystorage/schemas/............../tables/.............../part-00000-4838762b-144c-495b-9f63-2d7503d9db96.c000.snappy.parquet?st=2024-06-11T09%3A08%3A40Z&sv=2020-02-10&ske=2024-06-11T11%3A05%3A33Z&sig=KnHChlZneheFZlfMJvmTKyULCh67Noc7v2MV03bAjRY%3D&sktid=18a01ad8-9727-498a-a47d-17374c6fd9f7&se=2024-06-11T10%3A18%3A40Z&skoid=7c5198fc-d78c-4a43-a084-c6d7ad5dbf36&spr=https&sks=b&skt=2024-06-11T09%3A05%3A33Z&sp=r&skv=2020-10-02&sr=b

can someone suggest why is it erroring and how can I resolve?

Re: Delta sharing open protocol in Unity catalog: FileNotFoundError

jacovangelder — Tue, 11 Jun 2024 14:41:38 GMT

I wasn't able to reproduce your issue. Is your delta table operable? can you see sample data from within databricks and query the table from within databricks? It almost looks like some parquet files are missing, causing your delta not queryable anymore.

Re: Delta sharing open protocol in Unity catalog: FileNotFoundError

kiko_roy — Tue, 11 Jun 2024 16:00:45 GMT

yes My delta table is operable ,it has sample data and can be queried from sql-warehouse as well as notebooks. If I run the script from within databricks env (i.e same metastore) ,Data can be read with delta sharing. But If I run it outside from a jupyter notebook , unable to read .

Re: Delta sharing open protocol in Unity catalog: FileNotFoundError

jacovangelder — Tue, 11 Jun 2024 16:23:30 GMT

Then I am 100% sure that it is because your azure data lake storage does not have public network access enabled, or has a firewall or private endpoint setup. In order to query delta shares, you need to be able to access the storage account where the delta tables reside in.

Re: Delta sharing open protocol in Unity catalog: FileNotFoundError

jacovangelder — Tue, 11 Jun 2024 16:44:43 GMT

I just reproduced it for you. It is 100% the networking on the Azure storage account.

Re: Delta sharing open protocol in Unity catalog: FileNotFoundError

kiko_roy — Wed, 12 Jun 2024 11:11:06 GMT

Thanks @jacovangelder for checking this out. Even I was supposing so. I tested by running the code in a workspace under same metastore , and could read the data, and when ran the same code from a workspace of a different metastore had a same issue. So this test reconfirms. Surely my system is under private VNET without any endpoints created (as checked with my networking team later)