What is the bestway to handle huge gzipped file dropped to S3 ?

[RIDBX]

What is the bestway to handle huge gzipped file dropped to S3 ?=================================================

I find some intereting suggestions for posted questions. Thanks for reviewing my threads. Here is the situation we have.

We are getting data feed from on-prem to S3 , where datafeed is able to push data in only gzip format. When the dropped file are huge in s3 bucket/folder (eg north of 20GB). We are facing loading challenges in databricks. Our current databricks autoloader takes very long time with risk of retry upon failures of this load.

We know other databases have bulk load options with Parallel split/distribution  to handle this sutuation. 

Do we have such or better options in Databricks?

Are we able use databricks external table to tie S3 gizip file with partition option ? will this help ?

Do we need to go to upstraem on-prem data push process and ask for dropping files in smaller sizes (eg 4gb) into S3? 😂

I see the standard way for this decribed as 

"Read the Gzip File from S3: Use boto3 to read the gzip file from S3 and load it into your Databricks environment."

How did folks in this community addres these issues ?

Thanks for your guidance.

[Kaniz]

Hi @RIDBX, 

• One approach is to avoid using DataFrames and instead use RDDs (Resilient Distributed Datasets) for repartitioning.
• Read the gzipped files as RDDs, repartition them into smaller partitions, and save them in a splittable format (e.g., Snappy compression).
• Once the files are repartitioned, you can read the Snappy partitions as Spark DataFrames and process....
• Yes, you can use Databricks external tables to handle large files in S3.
• Create an external table that points to the gzipped files in S3. You can specify partition columns during table creation.
• Partitioning by relevant columns (e.g., date, region) can significantly improve query performance.
• Databricks will optimize queries by reading only the necessary partitions, reducing the load time.
• Remember to choose an appropriate partitioning strategy based on your data characteristics 
• Consider collaborating with the upstream on-prem data push process.
• Request that they split large files into smaller chunks (e.g., 4GB) before dropping them into S3.
• Smaller files are easier to manage, load, and process efficiently.
• This approach can help mitigate the challenges posed by large files
• The standard method involves using boto3, the AWS SDK for Python, to read the gzipped files from S3.
• Once you’ve read the files, load them into your Databricks environment for further processing.
• While this approach works, it’s essential to optimize the process based on your specific use case an...e.

[RIDBX]

@Kaniz 

Thanks for weighing in.

I learned that RDD is the predecessor to Dataframes.  What is the reason RDD perform better than Dataframes?

Are they using RDD for new implementations?

Thanks for patiently addressing my questions. tAre you able to tell us, what situation/condition each option applicable ?

Thanks for guidance.