Data Load from S3 Frankfurt Region to Unity catalog in AWS USWest Region
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-29-2026 02:29 AM
Hello,
I am trying to bring the Parquet data from S3 to the Unity Catalog
Currently I am doing straight forward read and write like this
test_data = spark.read.schema(1_billion_data).parquet(s3_path)
test_data.repartition(num_cores*2).write.mode("overwrite").saveAsTable(new_path)
Cluster is r5d.2xlarge
It takes roughly 1.45 hours to load this data, Is this normal ? can I bring this down to less than 20 minutes ?
Cluster is r5d.2xlarge
It takes roughly 1.45 hours to load this data, Is this normal ? can I bring this down to less than 20 minutes ?
