Hi @ambigus9,

Have you reviewed driver logs of the cluster, that would give us a clue on what the root of the issue is.

The security group that must have ports 443, 2443, 6666, 8443, 8444, 8445-8451 opened is the one created by Terraform, which is described as the "Data Plane VPC endpoint security group."

2) The destination for these ports should be 0.0.0.0/0, which allows traffic to any destination. This is necessary for the Databricks infrastructure, cloud data sources, library repositories, secure cluster connectivity, and other internal Databricks services

https://docs.databricks.com/en/security/network/classic/privatelink.html