How can I get Databricks notebooks to stop cutting off the explain plans?

User15787040559
Databricks Employee
Databricks Employee

‎06-07-2021 09:02 AM

(since Spark 3.0)

Dataset.queryExecution.debug.toFile

will dump the full plan to a file, without concatenating the output as a fully materialized Java string in memory.