topic Re: Capture events such as Start, Stop and Terminate of cluster. in Data Engineering

Capture events such as Start, Stop and Terminate of cluster.

sonalitotade — Wed, 18 Jan 2023 15:58:25 GMT

Hi,

I am using databricks with AWS.

I need to capture events such as Start, Stop and Terminate of cluster and perform some other action based on the events that happened on the cluster.

Is there a way I can achieve this in databricks?

daniel_sahal — Thu, 19 Jan 2023 07:00:14 GMT

@sonali totade

I see two ways of doing that:

Creating a custom app that will monitor Clusters API (list and get) and store their status somewhere, then trigger an action on new or status change,
Enable audit logging (https://docs.databricks.com/administration-guide/account-settings/audit-logs.html) and use CloudWatch or any other log analysis tool to create custom events.

sonalitotade — Tue, 31 Jan 2023 09:06:33 GMT

Hi Daniel,

thanks for the response

I would like to know if we can capture the event logs as shown in the image below when an event occurs on the cluster.