cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Send custom logs to AWS cloudwatch from Notebook

Murthy1
Contributor II

I would like to send some custom logs (in Python) from my Databricks notebook to AWS Cloudwatch.

For example:

df = spark.read.json(".......................")

logger.info("Successfully ingested data from json")

Has someone succeeded in doing this before? Your help is much appreciated. Thanks in advance!

5 REPLIES 5

Manoj12421
Valued Contributor II

Debayan
Esteemed Contributor III

Hi, You can integrate, please refer: https://aws.amazon.com/blogs/mt/how-to-monitor-databricks-with-amazon-cloudwatch/ and also you can configure audit logging to S3 and redirect it to cloudwatch from AWS. , refer: https://aws.amazon.com/blogs/mt/how-to-monitor-databricks-with-amazon-cloudwatch/

Hello @Debayan Mukherjee​  ,

Thanks for your reply and it was helpful.

To redirect log files from s3 to cloudwatch - I can use boto3 (for example :https://stackoverflow.com/questions/59147344/sending-emr-logs-to-cloudwatch)

However do we have any native integration with AWS cloudwatch?

Debayan
Esteemed Contributor III

Hi, the only native integration we have is : https://aws.amazon.com/blogs/mt/how-to-monitor-databricks-with-amazon-cloudwatch/.

Thank you!

Thanks for the Response!

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group