menotron
Valued Contributor

Yes, direct access to JVM is not allowed on Spark Connect or serverless compute. You could use the python logging framework to log into the output stream handle or any other Handler though.

import logging

logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)

stream_handler = logging.StreamHandler()
stream_handler.setLevel(logging.INFO)

formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
stream_handler.setFormatter(formatter)

logger.addHandler(stream_handler)