unzip twice the same file not executing

[RantoB]

Hi, 

I need to unzip some files that are ingested but when I unzip twice the same zipped file, the unzip command does not execute :

As suggesgted in the documentation I did :

import urllib 
urllib.request.urlretrieve("https://resources.lendingclub.com/LoanStats3a.csv.zip", "/tmp/LoanStats3a.csv.zip")

%sh
unzip /tmp/LoanStats3a.csv.zip

but when it apply again unzip, command never execute and seems to be blocked in a no out loop.

Thanks for you help.

[RantoB]

Ok.

Not that I have the same behaviour when I'm usig the python api :

with zipfile.ZipFile(path, 'r') as zip_ref:
    zip_ref.extractall(directory_to_extract_to)

[Anonymous]

If you're going to be reading the files with Spark, you don't need to unzip them. Spark's CSV reader can read zipped or unzipped CSVs.

If you're going to be using URL retrieve, remember that it will put the files on the driver and not in DBFS so you'll have to move it into the distributed filesystem to use Spark to read them.

[RantoB]

Actually, I will not be reading the file with SPark at this stage and I not using URL retrieve either, that was just for the reproductible example.

Zipped files are ingested on ADLS Gen2 and I unzip them into distinct directories depending on their names. But when I execute my script a second time, I am facing the problem I described above.

[Anonymous]

So when you use %sh it's going to use the file system on the driver, which is temporary. The driver storage is the local disc on a VM, not ADL2.

[RantoB]

Yes I understand but whatever I do with the unzipped files, I am asking why is there a problem executing twice the unzip action ?

[Hubert-Dudek]

1. Community edition can have blocked saving to file system or executing %sh
2. In other editions please verify that file there is. It can be saved for example to dbfs folder.

Following command could help:

dbutils.fs.ls("dbfs:/tmp/")
%sh
ls /dbfs/tmp

you can also consider to adjust your script to use dbfs prefix for example:

    import urllib 
    urllib.request.urlretrieve("https://resources.lendingclub.com/LoanStats3a.csv.zip", "/dbfs/tmp/LoanStats3a.csv.zip")

[RantoB]

I am trying to run :

with zipfile.ZipFile(path, 'r') as zip_ref:
    zip_ref.extractall(directory_to_extract_to)

but I think I am facing some issues because my zip file is quite large.

[jose_gonzalez]

hi @Bertrand BURCKER​ ,

Are you still having this issue or you were able to solve it? Please let us know.

[Atanu]

Could you please try without community edition, there must be some restriction for %sh

[RantoB]

No, still not solved.

[Hubert-Dudek]

what is you databricks version Azure or free community edition?

[jose_gonzalez]

Have you try the examples from this article? link

[RantoB]

I would like my code to be fully wwritten in python if possible.

[Atanu]

• Could you please try without community edition, there must be some restriction for %sh
•