Hi @Retired_mod,

Thanks for your prompt response.

1. I referred the volume information in the documentation.
2. Recommendations:
   • Number of files:
     • While I understand that having fewer files with larger sizes tends to perform better than having a larger number of files with smaller sizes, we've received a specific number of files from the upstream team.
     • These files are located in an S3 location designated for data processing.
     • My task now is to process this data and redistribute it for better performance.
   • Processing Efficiency:
     • In order to proceed with processing the data, I need to read it and form a data frame.
     • However, I'm encountering difficulties in reading the data efficiently, particularly due to its format and size.
   • I've opted to use an external volume to leverage the same storage for this purpose.
3. Alternatives:
   • The data is in the form of JSON files and is stored in an S3 bucket, not in ADLS Gen2.
   • Given these constraints, I'm exploring alternative approaches to efficiently read and process the data.
4. Use Case:
   1. To provide context, there are approximately 36 million JSON files in an S3 bucket that require processing.
   2. The objective is to create a delta table in the silver layer, which involves changing the file format and shuffling the data accurately.
   3. First to make it accessible in databricks, I've created an external volume on top of the folder containing all 36 million files.
   4. Now I'm trying to create a delta table in silver layer which changes the file format and shuffles the data correctly.
   5. To do the previous step, I'm using below spark line
      1.  spark.read.json(volume_path) where I'm encountering an error, as mentioned in yesterday's question.
   6. I'm seeking advice on whether there are alternative methods to read these files from the volume and create a data frame.

Your insights and guidance on this matter would be greatly appreciated.