Loading multiple gz files from ADLS to Delta Lake/Delta table in ADB

Krish1 — Fri, 28 Apr 2023 22:21:03 GMT

I have several gz files (file.csv.gz) in a ADLS folder which are of pretty big size. All of these files are extractd from the same base table so it has the similar data but of different dates. How can I transfer them in delta lake/delta table. We would like to do quick POC on how faster Databricks is in reading the data from these files in delta as other platform(s) couldn't scale it well.

Any help would be much appreciated.

Krish.

Re: Loading multiple gz files from ADLS to Delta Lake/Delta table in ADB

Debayan — Tue, 09 May 2023 13:03:24 GMT

Hi, You can read GZ files through spark.

https://stackoverflow.com/questions/42761912/how-to-read-gz-compressed-file-by-pyspark

Please let us know if this helps. Also, please tag @Debayan with your next comment so that I will get notified. Thank you!

topic Re: Loading multiple gz files from ADLS to Delta Lake/Delta table in ADB in Data Engineering

Loading multiple gz files from ADLS to Delta Lake/Delta table in ADB

Re: Loading multiple gz files from ADLS to Delta Lake/Delta table in ADB