topic Re: Spark read GZ file as corrupted data, when file extension having .GZ in upper case in Data Engineering

Spark read GZ file as corrupted data, when file extension having .GZ in upper case

hari-prasad — Fri, 19 Jan 2024 12:34:59 GMT

if file is renamed with file_name.sv.gz (lower case extension) is working fine, if file_name.sv.GZ (upper case extension) the data is read as corrupted, means it simply reading compressed file as is.

Re: Spark read GZ file as corrupted data, when file extension having .GZ in upper case

Lakshay — Fri, 19 Jan 2024 14:33:32 GMT

I don't think .GZ(upper case) is a valid file extension. I have seen most systems compress the file using .gz(lower case) extension

Re: Spark read GZ file as corrupted data, when file extension having .GZ in upper case

hari-prasad — Sun, 21 Jan 2024 07:05:12 GMT

I assume, if a .gz file is renamed as .GZ purposefully then we need to consider it as valid file format as gzip file. Cause that .GZ file still consist compressed data which is still valid.

Re: Spark read GZ file as corrupted data, when file extension having .GZ in upper case

Lakshay — Mon, 22 Jan 2024 15:57:23 GMT

Agree but Spark infers the compression from your filename and Spark cannot infer the compression from .GZ format. You can read more about this in below article:

https://aws.plainenglish.io/demystifying-apache-spark-quirks-2c91ba2d3978

Re: Spark read GZ file as corrupted data, when file extension having .GZ in upper case

hari-prasad — Mon, 22 Jan 2024 16:27:35 GMT

Yup, Spark does infer it from filename, I have been through spark code in Github.

Article is also refering to the internal code from Spark library.

I assume we can add an exception to handle .GZ file as gzip by tweaking spark libraries.

Re: Spark read GZ file as corrupted data, when file extension having .GZ in upper case

Lakshay — Mon, 22 Jan 2024 16:34:57 GMT

Yes, we can do it but is it worth doing it? This is something you can discuss in a Jira ticket.

Re: Spark read GZ file as corrupted data, when file extension having .GZ in upper case

hari-prasad — Mon, 22 Jan 2024 16:44:11 GMT

I assume it should worth handling such thing, as filename or extension should not be a constraint to process data.

As we know it's a gzip file and we can pass the paramter to read it as gzip.

Thanks a lot for your responses @Lakshay .

Re: Spark read GZ file as corrupted data, when file extension having .GZ in upper case

Lakshay — Mon, 22 Jan 2024 16:48:47 GMT

Happy to help!

Re: Spark read GZ file as corrupted data, when file extension having .GZ in upper case

hari-prasad — Sat, 10 Aug 2024 10:47:52 GMT

Recently I restarted look at a solution for this issue, I found out we can add few exception for allowing "GZ" in hadoop library as GzipCodec is invoked from there.