Databricks Community

acsmaggart · 10-31-2022

Background: I'm working on a pilot project to assess the pros and cons of using DataBricks to train models using R. I am using a dataset that occupies about 5.7GB of memory when loaded into a pandas dataframe. The data are stored in a delta table in ...

acsmaggart · 11-01-2022

Yes, the name of the dataframe was a little sloppy since it's a Pandas dataframe. Although about the scale, all of the machine learning documentation and sample ML notebooks for DataBricks that I have seen load the dataset into memory on the driver n...

acsmaggart · 11-01-2022

That is a good suggestion, and something I probably should have tried already. Although when I useSparkR::collectI get a JVM error:java.lang.OutOfMemoryError: Requested array size exceeds VM limit

Databricks Community

User Stats

User Activity

`collect()`ing Large Datasets in R

Re: `collect()`ing Large Datasets in R

Re: `collect()`ing Large Datasets in R