samshifflett46
Databricks Partner

Hey @ABINASH, The JSON file being flattened to 620 million records seems like the area of optimization would be to restructure the JSON file. My initial thought being that the JSON file is extremely nested which is causing a large amount of redundant information within the flattening. I would try finding the highest cardinality column in the flattened table and then extracting that data from the JSON file before flattening it. You could do this for a number of columns and simply have a foreign key between two tables that can be joined back together at a later period.

Could you present a snippet of the JSON file?