cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Monitoring

Anonymous
Not applicable

Are there any event streams that are or could be exposed in AWS (such as Cloudwatch Eventbridge events or SNS messages? In particular I'm interested in events that detail jobs being run. The use case here would be for monitoring jobs from our web application without needing to poll the jobs/list API endpoint.

1 REPLY 1

jessykoo32
New Contributor II

Yes, there are several event streams in AWS that can be used to monitor jobs being run. Your Texas Benefits

  1. CloudWatch Events: This service allows you to set up rules to automatically trigger actions in response to specific events in other AWS services, such as starting or stopping an EC2 instance, or an SNS message being published. You can create a custom event and set a rule to trigger when a job is run or completed.
  2. EventBridge: This is a serverless event bus that allows you to easily integrate different AWS services and your own applications. You can create custom events and set up rules to trigger when a job is run or completed.
  3. SNS (Simple Notification Service): This service allows you to send messages to one or more recipients, such as email, SMS, or other AWS services. You can set up an SNS topic and subscribe to it, so that you receive notifications when a job is run or completed.
  4. SQS (Simple Queue Service): This service allows you to store and retrieve messages between different systems and application. You can set up an SQS queue and send a message when a job is run or completed, which can be polled by your application to check the status of the job.
  5. Step functions: This service allows you to build and run distributed applications and microservices using visual workflows. Step Functions can emit events when a task is running or completed. This can be a useful way to track the progress of a job.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group