https://community.databricks.com/t5/data-engineering/after-several-iteration-of-filter-and-union-the-data-is-bigger/m-p/14962#M9367 <P>The process for me to build model is:</P><OL><LI>filter dataset and split into two datasets</LI><LI>fit model based on two datasets </LI><LI>union two datasets</LI><LI>repeat 1-3 steps</LI></OL><P>The problem is that after several iterations, the model fitting time becomes longer dramatically, and the I got error message: org.apache.spark.SparkException: Job aborted due to stage failure: Total size of serialized results of 9587 tasks (4.0 GB) is bigger than spark.driver.maxResultSize (4.0 GB). But in fact the data columns and rows stay the same. </P><P>As the model fitting time also increases, I don't think increasing spark.driver.maxResultSize will solve this problem. Any suggestion? Thanks.</P> Wed, 22 Sep 2021 19:36:52 GMT Geeya 2021-09-22T19:36:52Z https://community.databricks.com/t5/data-engineering/after-several-iteration-of-filter-and-union-the-data-is-bigger/m-p/14962#M9367 <P>The process for me to build model is:</P><OL><LI>filter dataset and split into two datasets</LI><LI>fit model based on two datasets </LI><LI>union two datasets</LI><LI>repeat 1-3 steps</LI></OL><P>The problem is that after several iterations, the model fitting time becomes longer dramatically, and the I got error message: org.apache.spark.SparkException: Job aborted due to stage failure: Total size of serialized results of 9587 tasks (4.0 GB) is bigger than spark.driver.maxResultSize (4.0 GB). But in fact the data columns and rows stay the same. </P><P>As the model fitting time also increases, I don't think increasing spark.driver.maxResultSize will solve this problem. Any suggestion? Thanks.</P> Wed, 22 Sep 2021 19:36:52 GMT https://community.databricks.com/t5/data-engineering/after-several-iteration-of-filter-and-union-the-data-is-bigger/m-p/14962#M9367 Geeya 2021-09-22T19:36:52Z https://community.databricks.com/t5/data-engineering/after-several-iteration-of-filter-and-union-the-data-is-bigger/m-p/14963#M9368 <P>I assume that you are using PySpark to train a model? It sounds like you are collecting data on the driver and likely need to increase the size. Can you share any code?</P> Wed, 22 Sep 2021 20:11:44 GMT https://community.databricks.com/t5/data-engineering/after-several-iteration-of-filter-and-union-the-data-is-bigger/m-p/14963#M9368 Ryan_Chynoweth 2021-09-22T20:11:44Z