topic Bug with display function with structs? in Machine Learning

Bug with display function with structs?

VVM — Fri, 16 Jun 2023 15:50:07 GMT

It appears to me that there's a deceptive bug when using the databricks display function and viewing struct data. For whatever reason, multiple spaces are cut down to only one:

from pyspark.sql.functions import struct, col
 
df = spark.createDataFrame([
  ("this has two  spaces", "this has three   spaces"),
  ("this has one space", "this has nospace")
], ["sc", "osc"])
 
df = df.select(struct(df.columns).alias("scstruct"))
 
display( df )

You'll see in the result that the values with 2, 3, and 4 spaces are cut down to single spaces.

I came across this while attempting to diagnose a regex -> due to this bug, I wasn't aware of what the data values actually were.

Re: Bug with display function with structs?

Anonymous — Sat, 17 Jun 2023 09:27:11 GMT

Hi @Patrick Mascari

Great to meet you, and thanks for your question!

Let's see if your peers in the community have an answer to your question. Thanks.

Re: Bug with display function with structs?

VVM — Mon, 19 Jun 2023 11:13:18 GMT

not so sure the community can help here as this appears to be a verifiable and reproducible Databricks bug?