How to enable custom metrics for databricks EC2 instance?

[Nandhini_Kumar]

Case 1: I have an AWS Databricks Instance, followed some steps for enabling CloudWatch for the Databricks EC2 instance. However, memory metrics is not available in the cloud watch. For that followed below steps for enabling custom metrics for the EC2 instance, facing some issue. 

1. Creat an IAM role with CloudWatchAgentServerPolicy and AmazonSSMManagedInstanceCore permission.
2. Attached above created IAM role to the Databricks EC2 instance.
3. Create Parameter for Linux machine as below data in value:

   ​{ "metrics": { "namespace": "Custom",

   "append_dimensions": {

    "InstanceId": "${aws:InstanceId}"

    },

    "metrics_collected": {

   "mem": {

    "measurement": [

   {

   "name": "mem_used_percent",

     "rename": "memory_used_percent"

   }

     ],

     "metrics_collection_interval": 60

     }  } } }
4. After 30 mins, run command on EC2 instance machine to install AmazonCloudWatchAgent
1. Systems Manager console > Run command > Run a command
2. Search AWS-ConfigureAWSPackage and selected that, then add AmazonCloudWatchAgent in the name field.
5. For the target selection, choose instance manually. Select the Databricks related EC2 instance.

Followed the documented steps, I've successfully completed the first four steps, including creating the IAM role, attaching it to the EC2 instance, and configuring the required parameters. However, upon reaching step 5, I encountered a challenge.

When attempting to install the AmazonCloudWatchAgent using the Systems Manager console and selecting the target instance manually, the Databricks-related EC2 instance is not showing up in the list. I've tried searching through the instance ID, but unfortunately, no instance ID associated with Databricks was found. 

While connecting to EC2 instance through ssh for checking if the instance installed the SSM or not, but in the databricks EC2 instance there is no key pair so unable to connect ssh for that instance. Need to way to enable custom metrics(memory metric) for the databricks EC2 instance.

Case 2: If the cluster is terminated means, the EC2 instance automatically moved to terminated state and if a cluster restarted then a new EC2 instance will created automatically. Then how can we enable a custom metrics if that process is continued, Is there any possible automated way for enabling custom metrics for all databricks related EC2 instance or provide any other way without we manually enable for each and every time it's terminated.

#Databricks EC2 instance custom metric enable

[Kaniz]

Hi @Nandhini_Kumar, 

• It’s unfortunate that the Databricks-related EC2 instance is not showing up in the list when you try to install the CloudWatch Agent manually.
• To resolve this, let’s ensure that the instance meets the following criteria:
  • The instance is running and accessible.
  • The instance has the necessary IAM permissions to interact with Systems Manager (SSM).
  • The instance is associated with the correct IAM role (which you’ve already created).
• Additionally, double-check that the instance is in the same AWS region where you’re trying to install the agent.
• You mentioned that there is no key pair associated with the Databricks EC2 instance, making it impossible to connect via SSH.
• To address this, consider the following options:
  • Instance Profile: If the instance has an instance profile (IAM role), you can use the Systems Manager Session Manager to access the instance without requiring SSH keys.
  • Temporary Key Pair: If possible, create a temporary key pair, associate it with the instance, and then connect via SSH. Remember to remove the key pair afterward for security reasons.
• Instead of using the Systems Manager console, consider using the AWS CLI or SDK to install the CloudWatch Agent on the EC2 instance. This approach provides more flexibility and control.
• You can use the aws ssm send-command command to execute the installation script remotely.
• Ensure that the IAM role attached to the EC2 instance has the necessary permissions to interact with the Systems Manager and install software.
• Specifically, check if the role has the AmazonSSMManagedInstanceCore policy attached.