I would like to send some custom logs (in Python) from my Databricks notebook to AWS Cloudwatch.

For example:

df = spark.read.json(".......................")

logger.info("Successfully ingested data from json")

Has someone succeeded in doing this before? Your help is much appreciated. Thanks in advance!