topic Re: rdd.foreachPartition() does nothing? in Data Engineering

rdd.foreachPartition() does nothing?

p42af — Wed, 09 Mar 2022 03:15:43 GMT

I expected the code below to print "hello" for each partition, and "world" for each record. But when I ran it the code ran but had no print outs of any kind. No errors either. What is happening here?

%scala
 
val rdd = spark.sparkContext.parallelize(Seq(1,2,3,4,5,6,7,8))
rdd.foreachPartition(partition => {
 println("hello")
 partition.foreach(fun=>{
   println("world")
 })
})

Re: rdd.foreachPartition() does nothing?

RKNutalapati — Wed, 09 Mar 2022 06:56:06 GMT

Check your stdout logs

Re: rdd.foreachPartition() does nothing?

Hubert-Dudek — Wed, 09 Mar 2022 10:49:34 GMT

Is it lazy evaluated so you need to trigger action I guess

Re: rdd.foreachPartition() does nothing?

jose_gonzalez — Wed, 06 Apr 2022 00:24:33 GMT

Hi @Leon P ,

You will need to check the executor's logs. Check the Stdout

Re: rdd.foreachPartition() does nothing?

jose_gonzalez — Mon, 25 Apr 2022 20:54:39 GMT

Hi @Leon P ,

Just a friendly follow-up. Do you still need help or you were able to check the executor's logs and find the messages?