cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

What are best instance types to use Delta Lake on AWS, Azure and GCP?

Srikanth_Gupta_
Valued Contributor

Best instance types to use Delta in a better way, are there any recommendations?

Example: i3.xlarge vs m5.2x large vs D3v2

2 REPLIES 2

sajith_appukutt
Honored Contributor II

Delta cache accelerated instance types (storage optimized, NVMe SSD) has delta cache enabled and configured by default . This includes Ls and Lsv2 in Azure and I3 in AWS. These instance types would be recommended for workloads where you'd be reading delta lake tables repeatedly as it'd accelerate data reads

In AWS c5d, r5d, and z1d series workers could be configured to use delta cache, though it is not enabled by default

Mooune_DBU
Valued Contributor

Depending on your queries, if you're looking for Delta Cache Optimized instances, here's the list per provider:

  • AWS: i3.* (i.e. i3.xlarge)
  • Azure: Ls-types (i.e. L4sv2)
  • GCP: n2-highmem-*
Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.