Hi @Chandan Angadi​ ,

Are you getting any other error or warning message? for example in your log4j or the std error logs?

I would also recommend to run your code with the default values. Without these settings:  

config <- spark_config()

  config$spark.sql.shuffle.partitions = 480

  config$spark.executor.cores = 5

  config$spark.executor.memory = "30G"

  config$spark.rpc.message.maxSize = 1945

  config$spark.executor.instances = 24

  config$spark.driver.memory = "30G"

  config$spark.sql.execution.arrow.sparkr.enabled = TRUE

  config$spark.driver.maxResultSize = 0

Just to narrow down and identify the the message happens with all the default values or not. Some of these Spark configs are not needed in Databricks, unless you want to fine tune your job. In this case we need to make sure your job runs fine, to have a reference point.