cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Spark Error : RScript (1243) terminated unexpectedly: Cannot call r___RBuffer__initialize().

chandan_a_v
Valued Contributor

grid_slice %>%

 sdf_copy_to(

  sc = sc,

  name = "grid_slice",

  overwrite = TRUE

 ) %>%

 sdf_repartition(

  partitions = min(n_executors * 3, NROW(grid_slice)),

  partition_by = "variable"

 ) %>%

 spark_apply(

  f = slice_data_wrapper,

  columns = c(

   variable = "character",

   max_slice = "integer",

   n_slices = "integer"

  ),

  context = list(

   metadata = metadata,

   s3_params = s3_params,

   subfolder = "target_data/orig"

  )

 ) %>%

 compute() %>%

 collect() 

I got below attached issue when I tried to execute the above code. grid_slice is a tibble.

Tried different version of arrow (1.x, 4.x, 6.x) but didn't work.

ERROR sparklyr: RScript (1243) terminated unexpectedly: Cannot call r___RBuffer__initialize(). See https://arrow.apache.org/docs/r/articles/install.html for help installing Arrow C++ libraries.

1 ACCEPTED SOLUTION

Accepted Solutions

chandan_a_v
Valued Contributor

Hi @Kaniz Fatma​ ,

 With R version 4.1.2, Spark version 2.4.5 and arrow version 5.0.0. The issue got fixed.

View solution in original post

8 REPLIES 8

Kaniz
Community Manager
Community Manager

Hi @Chandan Angadi​ ! My name is Kaniz, and I'm the technical moderator here. Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question first. Or else I will get back to you soon. Thanks.

rajatha_d
New Contributor II

Hi @Kaniz Fatma​ ,

I am also facing a similar issue, could you please provide the solution ASAP?

Thanks

chandan_a_v
Valued Contributor

Hi @Kaniz Fatma

Did you find any solution? Please let us know

Kaniz
Community Manager
Community Manager

Hi @Chandan Angadi​  and @Rajatha Dsouza​ , this could be due to your running an older version of R.

Can you please update the version and re-confirm?

chandan_a_v
Valued Contributor

Hi @Kaniz Fatma​ ,

Currently I am using R 3.6.2, I will upgrade it to 4.x and let you know. Thanks for the input.

Kaniz
Community Manager
Community Manager

Great.

chandan_a_v
Valued Contributor

Hi @Kaniz Fatma​ ,

 With R version 4.1.2, Spark version 2.4.5 and arrow version 5.0.0. The issue got fixed.

Kaniz
Community Manager
Community Manager

Hi @Chandan Angadi​ , That's awesome!

If that solves your answer, would you like to mark it as the best answer?

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.