topic Re: FileNotFoundError: [Errno 2] No such file or directory: '/FileStore/tables/flight_data.zip' The data and file exists in location mentioned above in Data Engineering

FileNotFoundError: [Errno 2] No such file or directory: '/FileStore/tables/flight_data.zip' The data and file exists in location mentioned above

Data_Engineer_3 — Wed, 27 Oct 2021 15:14:48 GMT

I am new to learning Spark and working on some practice; I have uploaded a zip file in DBFS /FileStore/tables directory and trying to run a python code to unzip the file; The python code is as:

from zipfile import *

with ZipFile("/FileStore/tables/flight_data.zip", "r") as zipObj:

zipObj.extractall()

It throws an error:

FileNotFoundError: [Errno 2] No such file or directory: '/FileStore/tables/flight_data.zip'

When i check manually and also through the code dbutils.fs.ls("/FileStore/tables/") it returns

Out[13]: [ FileInfo(path='dbfs:/FileStore/tables/flight_data.zip', name='flight_data.zip', size=59082358)]

Can someone please review and advise; I am using community edition to run this on cluster with configuration:

Data Bricks Runtime Version 8.3 (includes Apache Spark 3.1.1, Scala 2.12)

Re: FileNotFoundError: [Errno 2] No such file or directory: '/FileStore/tables/flight_data.zip' The data and file exists in location mentioned above

Hubert-Dudek — Wed, 27 Oct 2021 17:16:36 GMT

It is on dbfs mount so in most scenarios you should prefix everything with /dbfs (or dbfs:/ in databricks native functions, in many is not even needed as they handle only dbfs like dbutils). So please try:

from zipfile import *
 
with ZipFile("/dbfs/FileStore/tables/flight_data.zip", "r") as zipObj:
    zipObj.extractall()

Re: FileNotFoundError: [Errno 2] No such file or directory: '/FileStore/tables/flight_data.zip' The data and file exists in location mentioned above

Data_Engineer_3 — Wed, 27 Oct 2021 17:29:55 GMT

@Hubert Dudek

Hello Sir,

I tried in the way you suggested as well.However no luck! Still gives the same error.

Thank you,

Goutam

Re: FileNotFoundError: [Errno 2] No such file or directory: '/FileStore/tables/flight_data.zip' The data and file exists in location mentioned above

Hubert-Dudek — Wed, 27 Oct 2021 17:40:31 GMT

Do you have maybe high-concurrency server or some limited trial version (trial/free can make problem with reading with not native libraries).

Try also to explorer filesystem using shell commands by putting magic %sh in the first line in notebook to see is there /dbfs folder

%sh
ls /

Re: FileNotFoundError: [Errno 2] No such file or directory: '/FileStore/tables/flight_data.zip' The data and file exists in location mentioned above

Data_Engineer_3 — Wed, 27 Oct 2021 17:48:55 GMT

@Hubert Dudek

Hi Sir,

Working in community edition; Tried with magic commands as well.No luck! It says the command is not recognized.

Re: FileNotFoundError: [Errno 2] No such file or directory: '/FileStore/tables/flight_data.zip' The data and file exists in location mentioned above

Hubert-Dudek — Thu, 28 Oct 2021 08:50:47 GMT

so it seems that in community edition you can not direct access filesystem. You have access only to dbfs storage but you need to load there uncompressed object. So you need everywhere to prefix with dbfs:/ if it is not work for some function it will not work. As a last chance you can give a try like that:

    from zipfile import *
     
    with ZipFile("dbfs:/FileStore/tables/flight_data.zip", "r") as zipObj:
        zipObj.extractall()

Re: FileNotFoundError: [Errno 2] No such file or directory: '/FileStore/tables/flight_data.zip' The data and file exists in location mentioned above

Data_Engineer_3 — Thu, 28 Oct 2021 10:20:13 GMT

@Hubert Dudek

Hi Sir, No luck with this way also. :(.

Thank you for all the great suggestions though.😊

Re: FileNotFoundError: [Errno 2] No such file or directory: '/FileStore/tables/flight_data.zip' The data and file exists in location mentioned above

jose_gonzalez — Thu, 18 Nov 2021 17:55:27 GMT

Hi @Goutam Pal ,

Are you still having this issue? I think @Kaniz Fatma example will work great to solve your issue.

Re: FileNotFoundError: [Errno 2] No such file or directory: '/FileStore/tables/flight_data.zip' The data and file exists in location mentioned above

Data_Engineer_3 — Fri, 19 Nov 2021 03:10:44 GMT

@Jose Gonzalez @Kaniz Fatma : The issue still persists. Please find attached the screenshot of the error.

Thanks,

Goutam Pal

Re: FileNotFoundError: [Errno 2] No such file or directory: '/FileStore/tables/flight_data.zip' The data and file exists in location mentioned above

Anonymous — Fri, 19 Nov 2021 16:47:52 GMT

@Goutam Pal - Thank you for letting us know. I apologize about the inconvenience.

Re: FileNotFoundError: [Errno 2] No such file or directory: '/FileStore/tables/flight_data.zip' The data and file exists in location mentioned above

Data_Engineer_3 — Fri, 03 Dec 2021 18:51:30 GMT

Hello Kaniz..Will try and revert you back.

Re: FileNotFoundError: [Errno 2] No such file or directory: '/FileStore/tables/flight_data.zip' The data and file exists in location mentioned above

Mahima_Lalwani — Sat, 29 Jan 2022 19:20:50 GMT

I changed the Databricks runtime version in cluster and it worked in my case. Thank you @Kaniz Fatma

Re: FileNotFoundError: [Errno 2] No such file or directory: '/FileStore/tables/flight_data.zip' The

883022 — Wed, 19 Jul 2023 15:36:48 GMT

What if changing the runtime is not an option? I'm experiencing a similar issue using the following:

%pip install -r /dbfs/path/to/file.txt

This worked for a while, but now I'm getting the Errno 2 mentioned above. I am still able to print the same file using dbutils.fs.head('dbfs:/path/to/file.txt')