PrivateLink AWS - Databricks, "Cluster terminated. Reason: Security Daemon Registration Exception"

[MarSier]

Hi FerArribas,

I struggle with PrivateLink connection between Databricks account and my AWS account. I have seen that you had a similar problem. I can create a workspace, but when I try to create a cluster I get an error: "Cluster terminated. Reason: Security Daemon Registration Exception". Can you give me hints and details on how you solved this problem?

Thanks in advance.

[Anonymous · Accepted solution]

@Marcin Sieradzan​ :

The "Security Daemon Registration Exception" error occurs when the Databricks Security Agent running on the VPC can't register itself with the Databricks Control Plane. This error can happen due to a variety of reasons, such as incorrect network configuration or firewall rules.

Here are some steps to troubleshoot and resolve this issue:

1. Ensure that the Databricks Security Agent is running on the instances within the VPC that you want to connect to Databricks. You can check this by running the following command on an instance within the VPC: sudo systemctl status databricks-security-agent
2. Make sure that your VPC is properly configured with the correct route tables and network access control lists (NACLs) to allow traffic to and from the Databricks Control Plane. Check that you have created and attached a PrivateLink endpoint to your VPC.
3. Verify that the DNS resolution is working properly. You can do this by running the following command on an instance within the VPC: nslookup databricks-control-plane-aws-global.cloud.databricks.com. This command should return the Private IP address of the PrivateLink endpoint you have created.
4. Ensure that you have properly configured the security group associated with the Databricks workspace to allow traffic from the VPC.
5. Check that there are no firewall rules or network security rules that are blocking traffic from the VPC to the Databricks Control Plane.
6. Verify that you have the correct IAM roles and permissions set up in your AWS account to allow the necessary actions for PrivateLink. Check that the VPC endpoint policy allows access from the IAM role.
7. Finally, try creating the cluster again and monitor the logs for any error messages.

[Anonymous]

Hi @Marcin Sieradzan​ 

Thank you for posting your question in our community! We are happy to assist you.

To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your question?

This will also help other community members who may have similar questions in the future. Thank you for your participation and let us know if you need any further assistance!