Hello @ambigus9,

It seems like you are encountering issues with creating a compute cluster in a Databricks workspace configured with PrivateLink and a custom VPC using Terraform. The error message indicates that the Spark driver is becoming unresponsive on startup, which could be due to several reasons such as invalid Spark configurations, library conflicts, incorrect metastore configuration, or misconfigured init scripts.

Here are some steps you can take to troubleshoot and resolve the issue:

1. Check Spark Configurations and Init Scripts:
• Review the Spark configurations and ensure they are correctly set up. Invalid configurations can cause the driver to become unresponsive.
• Verify that the init scripts are correctly configured and do not contain errors that could prevent the Spark driver from starting.
2. Review Security Group Rules:
• Ensure that the security group rules for both inbound and outbound traffic are correctly configured. The necessary ports (443, 2443, 6666, 8443, 8444, 8445-8451) should be open as required by Databricks.
• Make sure that the security group allows traffic between the workspace subnets and the VPC endpoints.
3. Check VPC Endpoints:
• Verify that the VPC endpoints for the workspace and secure cluster connectivity relay are correctly set up and associated with the appropriate subnets and security groups.
• Ensure that the DNS hostnames and DNS resolution are enabled for the VPC.
4. Review Network ACLs:
• Ensure that the network ACLs for the subnets allow bidirectional (outbound and inbound) rules for the necessary ports.
5. Check AWS Service Endpoints:
• Ensure that the necessary AWS service endpoints (S3, STS, Kinesis) are correctly set up and accessible from the workspace subnets.
6. Review Spark Driver Logs:
• Access the Spark driver logs to get more detailed information about the error. The logs can provide insights into what might be causing the driver to become unresponsive.