We have recently been in the process of upgrading our Databricks clusters to version 11.3 LTS. As part of this upgrade, we have been working on integrating the logging of Spark events to LogAnalytics using the repository available at https://github.com/mspnp/spark-monitoring/tree/l4jv2.
However, we've encountered an issue where certain pieces of information that were previously available in azure log analytics table SparkLoggingEvent_CL in LTS 10.4 seem to be missing in the 11.3 LTS version. One specific example of this issue is related to the "Message" field. In LTS 10.4, the "Message" field used to contain driver log messages, but in our current setup with 11.3 LTS, this field appears to be populated with spaces, and none of the driver log messages are appearing in this field.
We are reaching out to kindly request your assistance in resolving this matter. It's crucial for us to accurately capture and analyze these driver log messages as they provide valuable insights into the behavior of our Spark applications. Could you please guide us on how to ensure that the driver log messages are correctly being logged and reflected in the "Message" field as they were in the previous version?
Any insights, recommendations, or troubleshooting steps you could provide would be greatly appreciated. We understand your expertise in this area and believe that your guidance will significantly help us in overcoming this challenge.
