topic Re: Send custom logs to AWS cloudwatch from Notebook in Data Engineering

Send custom logs to AWS cloudwatch from Notebook

Murthy1 — Wed, 08 Feb 2023 15:12:58 GMT

I would like to send some custom logs (in Python) from my Databricks notebook to AWS Cloudwatch.

For example:

df = spark.read.json(".......................")

logger.info("Successfully ingested data from json")

Has someone succeeded in doing this before? Your help is much appreciated. Thanks in advance!

Re: Send custom logs to AWS cloudwatch from Notebook

Manoj12421 — Thu, 09 Feb 2023 19:33:56 GMT

Refer the link:

https://docs.databricks.com/administration-guide/account-settings/audit-logs.html

Re: Send custom logs to AWS cloudwatch from Notebook

Murthy1 — Fri, 10 Feb 2023 09:52:11 GMT

Hello @Debayan Mukherjee ,

Thanks for your reply and it was helpful.

To redirect log files from s3 to cloudwatch - I can use boto3 (for example :https://stackoverflow.com/questions/59147344/sending-emr-logs-to-cloudwatch)

However do we have any native integration with AWS cloudwatch?

Re: Send custom logs to AWS cloudwatch from Notebook

Murthy1 — Mon, 13 Feb 2023 15:05:50 GMT

Thanks for the Response!

Re: Send custom logs to AWS cloudwatch from Notebook

Debayan — Fri, 10 Feb 2023 05:32:34 GMT

Hi, You can integrate, please refer: https://aws.amazon.com/blogs/mt/how-to-monitor-databricks-with-amazon-cloudwatch/ and also you can configure audit logging to S3 and redirect it to cloudwatch from AWS. , refer: https://aws.amazon.com/blogs/mt/how-to-monitor-databricks-with-amazon-cloudwatch/

Re: Send custom logs to AWS cloudwatch from Notebook

Debayan — Mon, 13 Feb 2023 05:39:56 GMT

Hi, the only native integration we have is : https://aws.amazon.com/blogs/mt/how-to-monitor-databricks-with-amazon-cloudwatch/.

Thank you!