cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

rdd.foreachPartition() does nothing?

p42af
New Contributor

I expected the code below to print "hello" for each partition, and "world" for each record. But when I ran it the code ran but had no print outs of any kind. No errors either. What is happening here?

%scala
 
val rdd = spark.sparkContext.parallelize(Seq(1,2,3,4,5,6,7,8))
rdd.foreachPartition(partition => {
 println("hello")
 partition.foreach(fun=>{
   println("world")
 })
})

1 ACCEPTED SOLUTION

Accepted Solutions

Hi @Leon P​ ,

You will need to check the executor's logs. Check the Stdout

View solution in original post

4 REPLIES 4

RKNutalapati
Valued Contributor

Check your stdout logs

Hubert-Dudek
Esteemed Contributor III

Is it lazy evaluated so you need to trigger action I guess

Hi @Leon P​ ,

You will need to check the executor's logs. Check the Stdout

Hi @Leon P​ ,

Just a friendly follow-up. Do you still need help or you were able to check the executor's logs and find the messages?

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.