cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Monitoring

Anonymous
Not applicable

Are there any event streams that are or could be exposed in AWS (such as Cloudwatch Eventbridge events or SNS messages? In particular I'm interested in events that detail jobs being run. The use case here would be for monitoring jobs from our web application without needing to poll the jobs/list API endpoint.

1 REPLY 1

jessykoo32
New Contributor II

Yes, there are several event streams in AWS that can be used to monitor jobs being run. Your Texas Benefits

  1. CloudWatch Events: This service allows you to set up rules to automatically trigger actions in response to specific events in other AWS services, such as starting or stopping an EC2 instance, or an SNS message being published. You can create a custom event and set a rule to trigger when a job is run or completed.
  2. EventBridge: This is a serverless event bus that allows you to easily integrate different AWS services and your own applications. You can create custom events and set up rules to trigger when a job is run or completed.
  3. SNS (Simple Notification Service): This service allows you to send messages to one or more recipients, such as email, SMS, or other AWS services. You can set up an SNS topic and subscribe to it, so that you receive notifications when a job is run or completed.
  4. SQS (Simple Queue Service): This service allows you to store and retrieve messages between different systems and application. You can set up an SQS queue and send a message when a job is run or completed, which can be polled by your application to check the status of the job.
  5. Step functions: This service allows you to build and run distributed applications and microservices using visual workflows. Step Functions can emit events when a task is running or completed. This can be a useful way to track the progress of a job.

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.