08-15-2023 08:00 AM - edited 08-17-2023 04:07 AM
We have recently been in the process of upgrading our Databricks clusters to version 11.3 LTS. As part of this upgrade, we have been working on integrating the logging of Spark events to LogAnalytics using the repository available at https://github.com/mspnp/spark-monitoring/tree/l4jv2.
However, we've encountered an issue where certain pieces of information that were previously available in azure log analytics table SparkLoggingEvent_CL in LTS 10.4 seem to be missing in the 11.3 LTS version. One specific example of this issue is related to the "Message" field. In LTS 10.4, the "Message" field used to contain driver log messages, but in our current setup with 11.3 LTS, this field appears to be populated with spaces, and none of the driver log messages are appearing in this field.
We are reaching out to kindly request your assistance in resolving this matter. It's crucial for us to accurately capture and analyze these driver log messages as they provide valuable insights into the behavior of our Spark applications. Could you please guide us on how to ensure that the driver log messages are correctly being logged and reflected in the "Message" field as they were in the previous version?
Any insights, recommendations, or troubleshooting steps you could provide would be greatly appreciated. We understand your expertise in this area and believe that your guidance will significantly help us in overcoming this challenge.
09-21-2023 01:05 PM
Hi Ravikumashi,
Can you please raise a ticket with us so that we can look deeper in to the issue
08-17-2023 04:03 AM
08-17-2023 04:33 AM
09-21-2023 01:05 PM
Hi Ravikumashi,
Can you please raise a ticket with us so that we can look deeper in to the issue
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.
If there isn’t a group near you, start one and help create a community that brings people together.
Request a New Group