Databricks compute failed to start due to EC2 security group id change in AWS default VPC

[sg-vtc]

I have been using Databricks on AWS for almost 2 years.  Recently, I cannot start up Databricks compute.  

      "databricks_error_message": "The VM launch request to AWS failed, please check your configuration. [details] InvalidGroup.NotFound: The security group 'sg-0ed732774bf1fxxxx' does not exist in VPC 'vpc-0b5c298399138xxxx' (Service: AmazonEC2; Status Code: 400; Error Code: InvalidGroup.NotFound; Request ID: 30805129-7513-4aad-b55d-7a1add373cc2; Proxy: null)(OnDemand)".

Checked my AWS account, the security group ID sg-0ed732774bf1fxxxx was deleted due to company compliance change.  A new security group is created in the default VPC vpc-0b5c298399138xxxx.  However, Databricks still trying to find the deleted security group. 

I did not configure any custom network configuration VPC when I created the workspace 2 years ago.  How to make Databricks uses the new security group in my AWS default VPC.   

[Isi]

hey @sg-vtc 

You need to register a new network configuration in Databricks that points to the updated VPC and security group. Here’s how:
- Go to the Databricks Account Console (not the workspace UI). (Click in workspaces dropbox and Manage Account)
- Navigate to: Manage Account → Cloud Resources → Network → Network Configurations
- Click “Add Network Configuration”
   · Add VPC ID, Subnets IDs, Security Group ID, and link with a workspace

Hope this helps 🙂

Isi