java.io.IOException: While processing file s3://test/abc/request_dt=2021-07-28/someParquetFile. [XYZ] BINARY is not in the store

Data Engineering

Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.

Hi Team,

I am facing an issue "java.io.IOException: While processing file s3://test/abc/request_dt=2021-07-28/someParquetFile. [XYZ] BINARY is not in the store"

The things i did before getting the above exception:

1. Alter table tableName1 add columns(xyz string);

2. ALTER TABLE tableName1 RECOVER PARTITIONS; -- Recovered the new partitioned equest_dt=2021-07-28

3. INSERT OVERWRITE TABLE tableName2

PARTITION (request_dt) SELECT old_column_names, XYZ from tableName1

And hence the the third command causes the Exception

Any suggestions would be of great help.

Thanks in Advance!!!