topic Re: Issue while trying to read a text file in databricks using Local File API's instead of Spark API. in Data Engineering

Issue while trying to read a text file in databricks using Local File API's instead of Spark API.

RiyazAliM — Wed, 24 Nov 2021 06:31:46 GMT

I'm trying to read a small txt file which is added as a table to the default db on Databricks. While trying to read the file via Local File API, I get a `FileNotFoundError`, but I'm able to read the same file as Spark RDD using SparkContext.

Please find the code below:

with open("/FileStore/tables/boringwords.txt", "r") as f_read:
  for line in f_read:
    print(line)

The error I get is:

FileNotFoundError                         Traceback (most recent call last)
<command-2618449717515592> in <module>
----> 1 with open("dbfs:/FileStore/tables/boringwords.txt", "r") as f_read:
      2   for line in f_read:
      3     print(line)
 
FileNotFoundError: [Errno 2] No such file or directory: 'dbfs:/FileStore/tables/boringwords.txt'

Where as, I have no problem reading the file using SparkContext:

boring_words = sc.textFile("/FileStore/tables/boringwords.txt")
set(i.strip() for i in boring_words.collect())

And as expected, I get the result for the above block of code:

Out[4]: {'mad',
 'mobile',
 'filename',
 'circle',
 'cookies',
 'immigration',
 'anticipated',
 'editorials',
 'review'}

I was also referring to the DBFS documentation to understand the Local File API's limitations but of no lead on the issue. Any help would be greatly appreciated. Thanks!

Re: Issue while trying to read a text file in databricks using Local File API's instead of Spark API.

-werners- — Wed, 24 Nov 2021 07:49:16 GMT

can you try with /dbfs/Filestore/tables/boringwords.txt?

Re: Issue while trying to read a text file in databricks using Local File API's instead of Spark API.

RiyazAliM — Wed, 24 Nov 2021 08:58:51 GMT

Hey there @Werner Stinckens ! Thanks for your response!

I've tried your suggestion and I still get the same error!

PFA the snip below:

Moreover, I've realized that adding ```/dbfs``` to the path is optional, as I've stored the data in the default database. Refer to the OP that I'm creating a RDD by passing the path ```"/FileStore/Tables/filename.txt"``` in `sc.textFile`.

Thank you!

Re: Issue while trying to read a text file in databricks using Local File API's instead of Spark API.

-werners- — Wed, 24 Nov 2021 09:11:48 GMT

you forgot a "/" as the first character in your file path.

Re: Issue while trying to read a text file in databricks using Local File API's instead of Spark API.

RiyazAliM — Wed, 24 Nov 2021 09:52:07 GMT

Hello @Werner Stinckens !

You're right! I missed the '/' earlier.

But, nothing changed after adding the '/' before dbfs. Below is the snip:

Moreover, when I tried the same path notation with SparkContext - It threw me an error:

I'm starting to wonder if this is the right way to provide the absolute path.

On the contrary, I've gave the path as "dbfs:/FileStore/tables/boringwords.txt" and it worked.

But again this doesn't work for reading the file from Local API.

Re: Issue while trying to read a text file in databricks using Local File API's instead of Spark API.

-werners- — Wed, 24 Nov 2021 10:08:06 GMT

No that should work.

I just tested it on my environment.

Also:

https://docs.microsoft.com/en-us/azure/databricks/data/databricks-file-system#python

https://community.databricks.com/s/question/0D53f00001HKHS7CAP/python-open-function-is-unable-to-detect-the-file-in-dbfs

But maybe you use the community edition of Databricks? If I recall correctly, the dbfs mounting is limited. So the local file interface might not work.

(See https://community.databricks.com/s/question/0D53f00001HKIFjCAP/where-is-dbfs-mounted-with-community-edition), not sure though.

If not: all I could think of is that the file is not there (so incorrect path), but SC can find it so that won't be it.

Proof it works:

Re: Issue while trying to read a text file in databricks using Local File API's instead of Spark API.

RiyazAliM — Wed, 24 Nov 2021 10:22:55 GMT

hey @Werner Stinckens ,

My apologies! Forgot to mention that I'm using the Databricks community edition. Thanks for the references, much appreciated!!

Re: Issue while trying to read a text file in databricks using Local File API's instead of Spark API.

RiyazAliM — Sun, 12 Dec 2021 16:00:55 GMT

Thank you for the help @Kaniz Fatma !! Appreciate it. 😀