Databricks CLI/SDKs not returning all logs even when less than 5 MB

[Kaz]

We're currently using the python sdk, but the same problem is in the databricks cli. The documentation states that when using workspace.jobs.get_run_output().logs, the last 5 MB of these logs are returned. However, we notice that the logs are truncated even when the logs are much smaller (we know because we log the same logs to a file as well). I'm looking for the actual reason logs get truncated. Number of lines? Line length? Hope somebody knows!

[Kaniz_Fatma]

Hi @Kaz, The truncation of logs in Databricks occurs when the size of the JSON representation exceeds 100 KB. When this limit is reached, the values are truncated, and the string “truncated” is appended to the affected entries. In rare cases where a truncated map is still larger than 100 KB, a single TRUNCATED key with an empt...¹. This behaviour applies to both the Python SDK and the Databricks CLI.

If you’re experiencing truncation even for smaller logs, it’s essential to check the overall size of the JSON representation of the logs. If the total size exceeds the specified limit, truncation will occur. You might want to investigate the specific content of the logs and their structure to identify any unusually large entries or patterns that contribute to the truncation.

Additionally, keep in mind that Databricks will only attempt to display the first 64,000 rows of data in dashboards. If the initial 64,000 rows are larger than 2,187 MB, it will fail to display anything². While this is related to dashboard display, it highlights the importance of managing log sizes to avoid truncation.

If you need further assistance or have additional questions, feel free to ask! 😊