topic Re: How do I download and unzip datasets from Kaggle into DBFS? in Data Engineering

How do I download and unzip datasets from Kaggle into DBFS?

StephanieAlba — Wed, 22 Mar 2023 14:30:08 GMT

Re: How do I download and unzip datasets from Kaggle into DBFS?

StephanieAlba — Wed, 22 Mar 2023 14:32:26 GMT

As shown here on StackOverflow

import opendatasets as od
 
od.download("https://www.kaggle.com/competitions/tlvmc-parkinsons-freezing-gait-prediction/data","/dbfs/FileStore/mypath/")

The output, when running this, shows first the zip being downloaded. Once the download is complete, it automatically extracts or unzips the files.

Extracting archive /dbfs/FileStore/mypath/tlvmc-parkinsons-freezing-gait-prediction/tlvmc-parkinsons-freezing-gait-prediction.zip to /dbfs/FileStore/mypath/tlvmc-parkinsons-freezing-gait-prediction

Re: How do I download and unzip datasets from Kaggle into DBFS?

pvignesh92 — Wed, 22 Mar 2023 15:22:30 GMT

Hi @Stephanie Rivera. In the Databricks notebook, you can handle this with either Python, Scala or bash.

I have not tried it but below should work as this command works in native shell.

%sh curl some_url --output myfile.zip
 
%sh unzip myfile.zip -d "some directory"

Re: How do I download and unzip datasets from Kaggle into DBFS?

karthik_p — Wed, 22 Mar 2023 16:14:10 GMT

@Stephanie Rivera please download u r kaggle file and unzip, if it is less than 100mb, you can follow below steps to directly to upload https://docs.databricks.com/ingestion/add-data/upload-data.html else please take below example and replace u r zip url and target

%sh curl https://resources.lendingclub.com/LoanStats3a.csv.zip --output /tmp/LoanStats3a.csv.zip

unzip /tmp/LoanStats3a.csv.zip

Re: How do I download and unzip datasets from Kaggle into DBFS?

Debayan — Thu, 23 Mar 2023 05:43:00 GMT

Hi, You can refer to https://docs.databricks.com/files/unzip-files.html. You can curl the file you want and then it can be unzipped as mentioned in the doc.

Please let us know if this helps.

Also, please tag @Debayan with your next update which will notify me.