Read data shared using Delta Sharing open sharing (for recipients) FileNotFoundError
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-25-2024 05:27 AM
I'm using the docs here: https://docs.databricks.com/en/data-sharing/read-data-open.html#store-creds
However I am unable to read the stored file which is sucessfully created with the following code:
%scala
dbutils.fs.put("dbfs:/FileStore/extraction/config.share","""
{"shareCredentialsVersion":1,"bearerToken":"XXXXXXXXXXXXXXXXXX","endpoint":"https://ukwest.azuredatabricks.net/api/2.0/delta-sharing/metastores/XXXXXXXXXXXXXXXXXX","expirationTime":"9999-12-31T23:59:59.999Z"}
""")
When I attempt to read back the file with the following example code:
import delta_sharing
client = delta_sharing.SharingClient(f"dbfs:/FileStore/extraction/config.share")
client.list_all_tables()
The following error is received:
FileNotFoundError: [Errno 2] No such file or directory: '/databricks/driver/dbfs:/FileStore/extraction/config.share'
File <command-3352291941134744>, line 2
1 import delta_sharing
----> 2 client = delta_sharing.SharingClient(f"dbfs:/FileStore/extraction/config.share")
3 client.list_all_tables()
File /local_disk0/.ephemeral_nfs/envs/pythonEnv-fe1622b9-cf42-4ef3-9ffc-a7ec8a5b3a2e/lib/python3.11/site-packages/fsspec/implementations/local.py:360, in LocalFileOpener._open(self)
358 if self.f is None or self.f.closed:
359 if self.autocommit or "w" not in self.mode:
--> 360 self.f = open(self.path, mode=self.mode)
361 if self.compression:
362 compress = compr[self.compression]
The file browser in Catalog, DBFS shows that the file is present.
What is going on here? Also why is the documentation incorrect?
Thanks in advance
Wayne
- Labels:
-
Spark
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-27-2024 12:33 AM
@WarmCat
As it states in the documentation, try reading the file from
/dbfs/<dbfs-path>/config.share
not from
dbfs:/<dbfs-path>/config.share
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-27-2024 03:01 AM
Thanks Daniel, but that does not work either.
The only thing that allowed progress was to use:
client = delta_sharing.SharingClient(f"file:///tmp/config.share")
I gave up and installed Apache Spark locally in a venv for now, and will be using AWS going forward.
Not a great experience with Databricks, for example even trying to preview this post results in an error!
Anyway, thanks for your efforts,
Wayne.

