Databricks

cralle · 08-01-2022

I have a DataFrame that I have created based on a couple of datasets and multiple operations. The DataFrame has multiple columns, one of which is a array of strings. But when I take the DataFrame and try to filter based upon the size of this array co...

cralle · 08-03-2022

@Werner Stinckens It worked by writing it and then reading it again. So I guess your theory of the query plan being too complex is true.

cralle · 08-02-2022

What do you mean by checkpointing?Also I have used display as I described in the question, I can do display/count directly before the filter and it works fine, but if I do it directly after the filter, then Spark just gets stuck and does nothing.

cralle · 08-02-2022

DBR 10.4 LTS. I run a lot of transformations and operations before I do this filter. But since I am able to display/count on the DataFrame before the filter, I would assume that it is because of the filter that the error is caused.

Databricks

User Stats

User Activity

Cannot display DataFrame when I filter by length

Re: Cannot display DataFrame when I filter by length

Re: Cannot display DataFrame when I filter by length

Re: Cannot display DataFrame when I filter by length