https://community.databricks.com/t5/data-engineering/why-a-new-project-instead-of-putting-this-in-apache-spark-itself/m-p/19946#M13452 <P>1. We want a venue in which we can rapidly iterate and make new releases. The overhead of making a release as a separate project is minuscule (in the order of minutes). A release on Spark takes a lot longer (in the order of days)</P><P>2. Koalas takes a different approach that might contradict Spark’s API design principles, and those principles cannot be changed lightly given the large user base of Spark. A new, separate project provides an opportunity for us to experiment with new design principles.</P> Fri, 25 Jun 2021 12:42:58 GMT User16826994223 2021-06-25T12:42:58Z https://community.databricks.com/t5/data-engineering/why-a-new-project-instead-of-putting-this-in-apache-spark-itself/m-p/19945#M13451 Fri, 25 Jun 2021 12:42:34 GMT https://community.databricks.com/t5/data-engineering/why-a-new-project-instead-of-putting-this-in-apache-spark-itself/m-p/19945#M13451 User16826994223 2021-06-25T12:42:34Z https://community.databricks.com/t5/data-engineering/why-a-new-project-instead-of-putting-this-in-apache-spark-itself/m-p/19946#M13452 <P>1. We want a venue in which we can rapidly iterate and make new releases. The overhead of making a release as a separate project is minuscule (in the order of minutes). A release on Spark takes a lot longer (in the order of days)</P><P>2. Koalas takes a different approach that might contradict Spark’s API design principles, and those principles cannot be changed lightly given the large user base of Spark. A new, separate project provides an opportunity for us to experiment with new design principles.</P> Fri, 25 Jun 2021 12:42:58 GMT https://community.databricks.com/t5/data-engineering/why-a-new-project-instead-of-putting-this-in-apache-spark-itself/m-p/19946#M13452 User16826994223 2021-06-25T12:42:58Z