AWS_INSUFFICIENT_INSTANCE_CAPACITY_FAILURE when starting SQL Server Ingestion pipeline

[kyeongmin_baek]

Dear Community,

I’m seeing a compute error when running a Databricks ingestion pipeline (Lakeflow managed ingestion) on AWS.

• Cloud : AWS

• Region: ap−northeast−2

• Source: SQL Server ingestion pipeline
  
  

  When I start the ingestion pipeline, it fails with the following error:

   

  { "reason": { "code": "AWS_INSUFFICIENT_INSTANCE_CAPACITY_FAILURE", "type": "CLIENT_ERROR", "parameters": { "databricks_error_message": "The VM launch failed due to insufficient capacity, please update your configuration and try again. [details] Unsupported: The requested configuration is currently not supported. Please check the documentation for supported configurations. (Service: AmazonEC2; Status Code: 400; Error Code: Unsupported; Request ID: 149f96aa-2904-4457-8fe3-6783eff7363f; Proxy: null)(OnDemand)", "instance_id": "failed-0ad8ca31-759c-404b-b", "aws_api_error_code": "Unsupported", "aws_error_message": "The requested configuration is currently not supported. Please check the documentation for supported configurations. (Service: AmazonEC2; Status Code: 400; Error Code: Unsupported; Request ID: 149f96aa-2904-4457-8fe3-6783eff7363f; Proxy: null)(OnDemand)" } }, "add_node_failure_details": { "failure_count": 2, "resource_type": "container", "will_retry": false } }

  I found that the job cluster on the page below is using an inappropriate instance type, and I currently suspect this is the cause of the issue.

  Could you help check why the ingestion pipeline cannot start in this workspace/region, and what configuration change or workaround you recommend?

  Thank you.

  

[Raman_Unifeye]

@kyeongmin_baek - I 'suspect' it is due to either because the instance type is not available in ap-northeast-2, or there’s temporary capacity exhaustion. This is common with On-Demand instances in less common regions or with large instance types.

To fix it, try changing the instance type in your pipeline configuration.