topic Re: What are best instance types to use Delta Lake on AWS, Azure and GCP? in Data Engineering

What are best instance types to use Delta Lake on AWS, Azure and GCP?

Srikanth_Gupta_ — Thu, 17 Jun 2021 17:22:03 GMT

Best instance types to use Delta in a better way, are there any recommendations?

Example: i3.xlarge vs m5.2x large vs D3v2

Re: What are best instance types to use Delta Lake on AWS, Azure and GCP?

sajith_appukutt — Thu, 17 Jun 2021 23:34:09 GMT

Delta cache accelerated instance types (storage optimized, NVMe SSD) has delta cache enabled and configured by default . This includes Ls and Lsv2 in Azure and I3 in AWS. These instance types would be recommended for workloads where you'd be reading delta lake tables repeatedly as it'd accelerate data reads

In AWS c5d, r5d, and z1d series workers could be configured to use delta cache, though it is not enabled by default

Re: What are best instance types to use Delta Lake on AWS, Azure and GCP?

Mooune_DBU — Thu, 17 Jun 2021 23:47:35 GMT

Depending on your queries, if you're looking for Delta Cache Optimized instances, here's the list per provider:

AWS: i3.* (i.e. i3.xlarge)
Azure: Ls-types (i.e. L4sv2)
GCP: n2-highmem-*