If you’re experiencing truncation even for smaller logs, it’s essential to check the overall size of the JSON representation of the logs. If the total size exceeds the specified limit, truncation will occur. You might want to investigate the specific content of the logs and their structure to identify any unusually large entries or patterns that contribute to the truncation.
Additionally, keep in mind that Databricks will only attempt to display the first 64,000 rows of data in dashboards. If the initial 64,000 rows are larger than 2,187 MB, it will fail to display anything2. While this is related to dashboard display, it highlights the importance of managing log sizes to avoid truncation.
If you need further assistance or have additional questions, feel free to ask! 😊