topic Re: unzip twice the same file not executing in Data Engineering

unzip twice the same file not executing

RantoB — Tue, 02 Nov 2021 15:17:44 GMT

Hi,

I need to unzip some files that are ingested but when I unzip twice the same zipped file, the unzip command does not execute :

As suggesgted in the documentation I did :

import urllib 
urllib.request.urlretrieve("https://resources.lendingclub.com/LoanStats3a.csv.zip", "/tmp/LoanStats3a.csv.zip")

%sh
unzip /tmp/LoanStats3a.csv.zip

but when it apply again unzip, command never execute and seems to be blocked in a no out loop.

Thanks for you help.

Re: unzip twice the same file not executing

RantoB — Tue, 02 Nov 2021 15:24:23 GMT

Ok.

Not that I have the same behaviour when I'm usig the python api :

with zipfile.ZipFile(path, 'r') as zip_ref:
    zip_ref.extractall(directory_to_extract_to)

Re: unzip twice the same file not executing

Anonymous — Tue, 02 Nov 2021 15:47:12 GMT

If you're going to be reading the files with Spark, you don't need to unzip them. Spark's CSV reader can read zipped or unzipped CSVs.

If you're going to be using URL retrieve, remember that it will put the files on the driver and not in DBFS so you'll have to move it into the distributed filesystem to use Spark to read them.

Re: unzip twice the same file not executing

RantoB — Tue, 02 Nov 2021 15:54:29 GMT

Actually, I will not be reading the file with SPark at this stage and I not using URL retrieve either, that was just for the reproductible example.

Zipped files are ingested on ADLS Gen2 and I unzip them into distinct directories depending on their names. But when I execute my script a second time, I am facing the problem I described above.

Re: unzip twice the same file not executing

Anonymous — Tue, 02 Nov 2021 15:58:05 GMT

So when you use %sh it's going to use the file system on the driver, which is temporary. The driver storage is the local disc on a VM, not ADL2.

Re: unzip twice the same file not executing

RantoB — Tue, 02 Nov 2021 16:02:31 GMT

Yes I understand but whatever I do with the unzipped files, I am asking why is there a problem executing twice the unzip action ?

Re: unzip twice the same file not executing

Hubert-Dudek — Tue, 02 Nov 2021 16:25:56 GMT

Community edition can have blocked saving to file system or executing %sh
In other editions please verify that file there is. It can be saved for example to dbfs folder.

Following command could help:

dbutils.fs.ls("dbfs:/tmp/")
%sh
ls /dbfs/tmp

you can also consider to adjust your script to use dbfs prefix for example:

    import urllib 
    urllib.request.urlretrieve("https://resources.lendingclub.com/LoanStats3a.csv.zip", "/dbfs/tmp/LoanStats3a.csv.zip")

Re: unzip twice the same file not executing

RantoB — Tue, 02 Nov 2021 17:53:24 GMT

I am trying to run :

with zipfile.ZipFile(path, 'r') as zip_ref:
    zip_ref.extractall(directory_to_extract_to)

but I think I am facing some issues because my zip file is quite large.

Re: unzip twice the same file not executing

jose_gonzalez — Tue, 16 Nov 2021 00:23:33 GMT

hi @Bertrand BURCKER ,

Are you still having this issue or you were able to solve it? Please let us know.

Re: unzip twice the same file not executing

RantoB — Tue, 16 Nov 2021 08:32:55 GMT

No, still not solved.

Re: unzip twice the same file not executing

Prabakar — Tue, 16 Nov 2021 16:39:05 GMT

Hi @Bertrand BURCKER as you have mentioned your zip file is large, can you let us know the size of the file?

Also, have you tried with a smaller zip file, and what is the result?

Re: unzip twice the same file not executing

Hubert-Dudek — Tue, 16 Nov 2021 18:04:08 GMT

what is you databricks version Azure or free community edition?

Re: unzip twice the same file not executing

jose_gonzalez — Mon, 22 Nov 2021 22:45:46 GMT

Have you try the examples from this article? link

Re: unzip twice the same file not executing

RantoB — Fri, 26 Nov 2021 18:09:49 GMT

My file is 180MiB. For information, the culster is a single node standard_F4s

Re: unzip twice the same file not executing

RantoB — Fri, 26 Nov 2021 18:26:21 GMT

I would like my code to be fully wwritten in python if possible.

Re: unzip twice the same file not executing

Atanu — Tue, 30 Nov 2021 06:54:43 GMT

Could you please try without community edition, there must be some restriction for %sh

Re: unzip twice the same file not executing

Atanu — Tue, 30 Nov 2021 06:54:54 GMT

Could you please try without community edition, there must be some restriction for %sh

Re: unzip twice the same file not executing

Hubert-Dudek — Wed, 15 Dec 2021 13:39:44 GMT

Another problem is that dbfs storage doesn't support random writes (used by zip):

Does not support random writes. For workloads that require random writes, perform the operations on local disk first and then copy the result to

/dbfs

source: https://docs.databricks.com/data/databricks-file-system.html#local-file-api-limitations