topic Re: How to avoid trimming in EXPLAIN? in Data Engineering

How to avoid trimming in EXPLAIN?

vr — Sun, 27 Nov 2022 00:03:04 GMT

I am looking on EXPLAIN EXTENDED plan for a statement.

In == Physical Plan == section, I go down to FileScan node and see a lot of ellipsis, like

                     +- FileScan parquet schema.table[Time#8459,TagName#8460,Value#8461,Quality#8462,day#8466,isLate#8467] Batched: true, DataFilters: [isnotnull(TagName#8460), isnotnull(Quality#8462), isnotnull(Value#8461), isnotnull(Time#8459), (..., Format: Parquet, Location: PreparedDeltaFileIndex(1 paths)[mcfs-abfss://t-125a3c9d-90a3-46dc-a577-196577aff13d+abc-masked..., PartitionFilters: [isnotnull(day#8466), (cast(day#8466 as timestamp) >= 2022-11-19 00:00:00), (cast(day#8466 as tim..., PushedFilters: [IsNotNull(TagName), IsNotNull(Quality), IsNotNull(Value), IsNotNull(Time), EqualTo(TagName,FRO_P..., ReadSchema: struct<Time:timestamp,TagName:string,Value:double,Quality:int>

How to see full description of the section, without trimming? I am particularly interested in PartitionFilters section.

Re: How to avoid trimming in EXPLAIN?

Anonymous — Sun, 27 Nov 2022 06:26:17 GMT

Hi @Vladimir Ryabtsev

Great to meet you, and thanks for your question!

Let's see if your peers in the community have an answer to your question first. Or else bricksters will get back to you soon.

Thanks

Re: How to avoid trimming in EXPLAIN?

-werners- — Mon, 28 Nov 2022 16:08:57 GMT

You can try using FORMATTED as display option, this does not truncate. But it will probably not display everything you want.

There are also the following parameters:

spark.sql.maxMetadataStringLength

spark.sql.maxPlanStringLength

I'd try changing the default values and see if it works.

https://spark.apache.org/docs/latest/configuration.html

Re: How to avoid trimming in EXPLAIN?

SS2 — Tue, 29 Nov 2022 19:09:23 GMT

I also faced the same

Re: How to avoid trimming in EXPLAIN?

vr — Tue, 29 Nov 2022 21:39:58 GMT

Indeed, FORMATTED gave better results out of the box.

Indeed it did not have everything, but tweaking spark.sql.maxMetadataStringLength helped with EXTENDED!

I did not quite get meaning of spark.sql.maxPlanStringLength, in my cluster it defaults to a strange number "2147483632b".

Re: How to avoid trimming in EXPLAIN?

UmaMahesh1 — Fri, 02 Dec 2022 18:09:40 GMT

Hi @Vladimir Ryabtsev

Glad you found a solution.

That strange number you are seeing is the number of bits.

maxPlanStringLength is used to set the max number of characters (default = 2147483632) we can output to a plan string. Anything more than that, output will be truncated. Ideally this value is large enough for you to print out the plan using formatted and other options. But sometimes, using other options to extend the output from it's truncated state sometimes leads to OutOfMemory errors in the driver node or processes.

Cheers.