How to work with 300 billions rows and 5 columns?

[guilhermecs001]

Hi guys!

I'm having a problem at work where I need to process a customer data dataset with 300 billion rows and 5 columns. The transformations I need to perform are "simple," like joins to assign characteristics to customers. And at the end of the process, I need to save a .csv file to S3. Currently, my notebook takes 35 to 50 hours to run. The data volume is really huge, we have over 100 million customers.

[szymon_dybczak]

Hi @guilhermecs001 ,

Wow, that's massive amount of rows. Can you somehow preprocess first this huge CSV file? For example, read CSV, partition by some columns that makes sense (maybe country from which customer is coming from) and save that data as delta format?
And then you will have a lot more option to optimize because delta gives you things like partition pruning, column pruning etc. 
So once you will have preprocessed data as Delta, you can try apply your join logic and save it as .CSV to S3 bucket.