topic Re: File not found error. in Data Governance

File not found error.

Data_Analytics1 — Wed, 22 Feb 2023 04:46:33 GMT

FileNotFoundError: [Errno 2] No such file or directory: '/dbfs/FileStore/config.share'

When I am trying to read the config.share file, it is throwing this error. I tried with spark path format as well which is dbfs:/FileStore/config.share' but it also didn't work.

Cluster configuration: 11.3 LTS (includes Apache Spark 3.3.0, Scala 2.12) Unity Catalog enabled

Re: File not found error.

Data_Analytics1 — Wed, 22 Feb 2023 13:40:22 GMT

Hi @Kaniz Fatma File is there even it shows full details using dbutils.fs.ls("/FileStore/")

but when I give the same path to delta sharing library, it gives me this error.

import delta_sharing
profile_file = f"/dbfs/FileStore/config.share"
profile = delta_sharing.SharingClient(profile_file)

When executing the above code snippet, FileNotFound error pops up.

Re: File not found error.

Data_Analytics1 — Wed, 22 Feb 2023 16:07:07 GMT

Hi @Kaniz Fatma I can read the same file in below code snippet.

table_url_spark = "dbfs:/FileStore/shares/config.share" + "#share.schema.table"
data_spark_cdf = delta_sharing.load_as_spark(table_url_spark)
data_spark_cdf.count()

But with the same path I am unable to read the file in below code snippet.

import delta_sharing
profile_file = f"dbfs:/FileStore/share/config.share"
profile = delta_sharing.SharingClient(profile_file)

Re: File not found error.

Data_Analytics1 — Wed, 22 Feb 2023 16:38:33 GMT

Hi @Kaniz Fatma , By mistake while copy pasting the code in the comment that extra 's' was deleted from shares word. But the problem still exist.

Re: File not found error.

pfig — Sun, 19 Mar 2023 22:38:06 GMT

Something is going on wth DBFS files/paths. I have experienced similar situations, even with simple files and paths. No matter the path format you decide to use; it's not working.

Re: File not found error.

JanaP — Tue, 11 Jul 2023 11:29:20 GMT

I receive the "File not found error" when reading dbfs file after the cluster was changed. Just with the same file and it's dbfs location there is no error when I change the cluster back.

Re: File not found error.

raghu2 — Tue, 09 Apr 2024 21:18:50 GMT

I have the same issue on a UC enabled cluster Db Version: 14.3 LTS. When I switch to cluster version 10.X there is no error.

Re: File not found error.

raghu2 — Thu, 11 Apr 2024 12:09:59 GMT

Was able to resolve the issue by adding pyOpenSSL library to the cluster.

Re: File not found error.

qigang — Thu, 30 May 2024 07:25:47 GMT

Have you solved?i have same problem

Re: File not found error.

dkushari — Thu, 30 May 2024 11:37:15 GMT

Few things to consider here - Cluster type, DBR version. Can you please check if the cluster is shared or assigned? This link might be helpful - https://docs.databricks.com/en/dbfs/unity-catalog.html

Re: File not found error.

jacovangelder — Wed, 12 Jun 2024 09:14:44 GMT

On Unity Catalog Shared Access Mode clusters you need to use a UC Volume to read (config) files using vanilla Python (with open() for example that many libs use). You can no longer read files from DBFS this way. This is all part of the new security model.

So you would get:

profile_file = f"/Volumes/Volume_123/share/config.share"

Good luck!