How can I get Databricks notebooks to stop cutting off the explain plans?
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-07-2021 09:02 AM
(since Spark 3.0)
Dataset.queryExecution.debug.toFilewill dump the full plan to a file, without concatenating the output as a fully materialized Java string in memory.
Labels:
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-28-2021 12:16 PM
Notebooks really aren't the best method of viewing large files. Two methods you could employ are
- Save the file to dbfs and then use databricks CLI to download the file
- Use the web terminal
In the web terminal option you can do something like "cat my_large_file.txt | less" and then you can scroll through the entire content of the file.
