05-06-2022 02:15 AM
Hi,
I need to ingest 60 millions json files from S3 and have create a Delta Live Tables to ingest these data to delta table with Auto Loader. However the input rate in my DLT is always around 8 records/second no matter how many worker I add to the DLT. I'm using all the default setting and have set the DLT to production mode. Is there any config that I need to add to increase the input rate for my DLT?
Thanks,
05-06-2022 09:01 AM
Please consider the following:
05-06-2022 09:01 AM
Please consider the following:
05-07-2022 06:11 AM
Hi Hubert,
Thank you very much for your answer. I have try your suggestion and have some follow up questions that I post inline with your answer here.
If you can help with these follow up questions that would be greatly appreciated. I'm very new to the Spark/Databricks and data analysis field in general so trying my best to learn here.
Thanks
05-07-2022 09:38 AM
Regarding private VPC yes that's the link https://docs.databricks.com/administration-guide/cloud-configurations/aws/customer-managed-vpc.html?
Regarding region is easier when you set up databricks you choose region and availability zone, the same for each S3 bucket. Just make sure that it is the same for both. For example us-west-2 etc.
05-07-2022 09:38 PM
Thanks, I'll follow the guide to setup the VPC.
Regarding the S3 transfer acceleration, do you know how to connect to it from Databricks without getting the "IllegalStateException: To enable accelerate mode, please use AmazonS3ClientBuilder.withAccelerateModeEnabled(true)" error?
06-05-2022 09:46 PM
Hi @thanh nguyen, Here is an excellent explanation of the issue faced you.
Please have a look.
06-14-2022 09:13 AM
Hi @thanh nguyen , We haven’t heard from you on the last response from me, and I was checking back to see if you have a resolution yet. If you have any solution, please share it with the community as it can be helpful to others. Otherwise, we will respond with more details and try to help.
Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections.
Click here to register and join today!
Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.