Case 1: I have an AWS Databricks Instance, followed some steps for enabling CloudWatch for the Databricks EC2 instance. However, memory metrics is not available in the cloud watch. For that followed below steps for enabling custom metrics for the EC2 instance, facing some issue.
Creat an IAM role with CloudWatchAgentServerPolicy and AmazonSSMManagedInstanceCore permission.
Attached above created IAM role to the Databricks EC2 instance.
Create Parameter for Linux machine as below data in value:
{ "metrics": { "namespace": "Custom",
"append_dimensions": {
"InstanceId": "${aws:InstanceId}"
},
"metrics_collected": {
"mem": {
"measurement": [
{
"name": "mem_used_percent",
"rename": "memory_used_percent"
}
],
"metrics_collection_interval": 60
} } } }
After 30 mins, run command on EC2 instance machine to install AmazonCloudWatchAgent
Systems Manager console > Run command > Run a command
Search AWS-ConfigureAWSPackage and selected that, then add AmazonCloudWatchAgent in the name field.
For the target selection, choose instance manually. Select the Databricks related EC2 instance.
Followed the documented steps, I've successfully completed the first four steps, including creating the IAM role, attaching it to the EC2 instance, and configuring the required parameters. However, upon reaching step 5, I encountered a challenge.
When attempting to install the AmazonCloudWatchAgent using the Systems Manager console and selecting the target instance manually, the Databricks-related EC2 instance is not showing up in the list. I've tried searching through the instance ID, but unfortunately, no instance ID associated with Databricks was found.
While connecting to EC2 instance through ssh for checking if the instance installed the SSM or not, but in the databricks EC2 instance there is no key pair so unable to connect ssh for that instance. Need to way to enable custom metrics(memory metric) for the databricks EC2 instance.
Case 2: If the cluster is terminated means, the EC2 instance automatically moved to terminated state and if a cluster restarted then a new EC2 instance will created automatically. Then how can we enable a custom metrics if that process is continued, Is there any possible automated way for enabling custom metrics for all databricks related EC2 instance or provide any other way without we manually enable for each and every time it's terminated.
#Databricks EC2 instance custom metric enable
