Send custom logs to AWS cloudwatch from Notebook

[Murthy1]

I would like to send some custom logs (in Python) from my Databricks notebook to AWS Cloudwatch.

For example:

df = spark.read.json(".......................")

logger.info("Successfully ingested data from json")

Has someone succeeded in doing this before? Your help is much appreciated. Thanks in advance!

[Manoj12421]

​Refer the link:

https://docs.databricks.com/administration-guide/account-settings/audit-logs.html

[Debayan]

Hi, You can integrate, please refer: https://aws.amazon.com/blogs/mt/how-to-monitor-databricks-with-amazon-cloudwatch/ and also you can configure audit logging to S3 and redirect it to cloudwatch from AWS. , refer: https://aws.amazon.com/blogs/mt/how-to-monitor-databricks-with-amazon-cloudwatch/

[Murthy1]

Hello @Debayan Mukherjee​  ,

Thanks for your reply and it was helpful.

To redirect log files from s3 to cloudwatch - I can use boto3 (for example :https://stackoverflow.com/questions/59147344/sending-emr-logs-to-cloudwatch)

However do we have any native integration with AWS cloudwatch?

[Debayan]

Hi, the only native integration we have is : https://aws.amazon.com/blogs/mt/how-to-monitor-databricks-with-amazon-cloudwatch/.

Thank you!

[Murthy1]

Thanks for the Response!