rdd.foreachPartition() does nothing?

p42af
New Contributor

I expected the code below to print "hello" for each partition, and "world" for each record. But when I ran it the code ran but had no print outs of any kind. No errors either. What is happening here?

%scala
 
val rdd = spark.sparkContext.parallelize(Seq(1,2,3,4,5,6,7,8))
rdd.foreachPartition(partition => {
 println("hello")
 partition.foreach(fun=>{
   println("world")
 })
})

RKNutalapati
Valued Contributor

Check your stdout logs

Hubert-Dudek
Databricks MVP

Is it lazy evaluated so you need to trigger action I guess


My blog: https://databrickster.medium.com/

Hi @Leon P​ ,

You will need to check the executor's logs. Check the Stdout

View solution in original post

Hi @Leon P​ ,

Just a friendly follow-up. Do you still need help or you were able to check the executor's logs and find the messages?