Spark read GZ file as corrupted data, when file extension having .GZ in upper case

[hprasad]

if file is renamed with file_name.sv.gz (lower case extension) is working fine, if file_name.sv.GZ (upper case extension) the data is read as corrupted, means it simply reading compressed file as is.

[Lakshay]

I don't think .GZ(upper case) is a valid file extension. I have seen most systems compress the file using .gz(lower case) extension

[hprasad]

I assume, if a .gz file is renamed as .GZ purposefully then we need to consider it as valid file format as gzip file. Cause that .GZ file still consist compressed data which is still valid.

[Lakshay]

Agree but Spark infers the compression from your filename and Spark cannot infer the compression from .GZ format. You can read more about this in below article:

https://aws.plainenglish.io/demystifying-apache-spark-quirks-2c91ba2d3978

[hprasad]

Yup, Spark does infer it from filename, I have been through spark code in Github.

Article is also refering to the internal code from Spark library. 

I assume we can add an exception to handle .GZ file as gzip by tweaking spark libraries. 

[Lakshay]

Yes, we can do it but is it worth doing it? This is something you can discuss in a Jira ticket. 

[hprasad]

I assume it should worth handling such thing, as filename or extension should not be a constraint to process data.

As we know it's a gzip file and we can pass the paramter to read it as gzip.

Thanks a lot for your responses @Lakshay . 

[Lakshay]

Happy to help!

[hprasad]

Recently I restarted look at a solution for this issue, I found out we can add few exception for allowing "GZ" in hadoop library as GzipCodec is invoked from there.