- 2407 Views
- 3 replies
- 5 kudos
Resolved! Understand Spark DataFrames verse R DataFrames
CommunityI’ve been struggling with utilizing R language in databricks and after reading “Mastering Spark with R,” I believe my initial problems stemmed from not understating the difference between Spark DataFrames and R DataFrames within the databric...
- 2407 Views
- 3 replies
- 5 kudos
Latest Reply
- 5 kudos
As Spark dataframes are handled in distributed way on workers it is better just to use Spark dataframes. Additionally collect is executed on driver and takes whole dataset into memory so it is shouldn't be used in production.
- 5 kudos