topic Re: FileNotFoundError while reading PDF file in Databricks from DBFS location in Data Engineering

FileNotFoundError while reading PDF file in Databricks from DBFS location

sahil07 — Tue, 27 Aug 2024 15:53:19 GMT

I am trying to read a PDF file from DBFS location in Databricks using PyPDF2.PdfFileReader but it's throwing error that file doesn't exist

But the file exists in the path, refer below screenshot

Can anyone please suggest what is wrong in this?

Re: FileNotFoundError while reading PDF file in Databricks from DBFS location

Lucas_TBrabo — Tue, 27 Aug 2024 17:40:15 GMT

Hi @sahil07!

As you are reading using PyPDF2, which does not use the spark API to read data, you should use "/dbfs/FileStore/sahil_chowdhurry.pdf" instead of "dbfs:/FileStore/sahil_chowdhurry.pdf".

As a general rule of thumb: If you are using readers that talks with the spark API, use the "dbfs:/", otherwise, use "/dbfs/".

Test it and let me know if it worked 🙂

Re: FileNotFoundError while reading PDF file in Databricks from DBFS location

sahil07 — Tue, 27 Aug 2024 18:03:48 GMT

Hi @Lucas_TBrabo

Used the one you suggested but same issue

Re: FileNotFoundError while reading PDF file in Databricks from DBFS location

Lucas_TBrabo — Tue, 27 Aug 2024 19:25:59 GMT

@sahil07 are you running this in a serverless cluster? If not, please let me know the config and runtime, please.

Re: FileNotFoundError while reading PDF file in Databricks from DBFS location

sahil07 — Tue, 27 Aug 2024 19:31:34 GMT

@Lucas_TBrabo I am using databricks community edition, DBR 14.3 LTS Spark 3.5.0 Scala 2.12

Re: FileNotFoundError while reading PDF file in Databricks from DBFS location

Lucas_TBrabo — Tue, 27 Aug 2024 20:36:19 GMT

@sahil07, It seems that with your current setup, you can't read from DBFS using vanilla Python. I've ran some tests and managed to reproduce the error and solve it by copying the file in DBFS to the local file system of the driver node using dbutils.fs.cp to copy to "file:/"

Try the following:

Re: FileNotFoundError while reading PDF file in Databricks from DBFS location

sahil07 — Wed, 28 Aug 2024 02:23:33 GMT

Thanks a lot @Lucas_TBrabo it worked.

But I am just wondering when I was trying to read csv files utilising the same cluster configs and using spark.read.csv() , I was able to read it without any issues. So, is it something related to PDF files that we can't directly read it from DBFS? And if yes then what kind of cluster configs is required to read PDF files directly from DBFS?

Re: FileNotFoundError while reading PDF file in Databricks from DBFS location

Lucas_TBrabo — Wed, 28 Aug 2024 12:27:51 GMT

@sahil07, the fact that you could read a csv file using spark.read.csv() is because you're using the spark native API to access the dbfs, which works just fine. The PDF reading was not possible because PyPDF2 does not use the spark API, but python standard reader.

Re: FileNotFoundError while reading PDF file in Databricks from DBFS location

sahil07 — Wed, 28 Aug 2024 16:08:05 GMT

@Lucas_TBrabo thanks for the detailed explanation, really appreciate it.