topic How to perform Inner join using withcolumn in Data Engineering

How to perform Inner join using withcolumn

pramalin — Mon, 30 Jan 2023 18:33:47 GMT

Re: How to perform Inner join using withcolumn

daniel_sahal — Tue, 31 Jan 2023 07:59:18 GMT

@prudhvi ramalingam

Here is example: https://stackoverflow.com/a/61029482

Re: How to perform Inner join using withcolumn

Nhan_Nguyen — Tue, 31 Jan 2023 13:19:44 GMT

@prudhvi ramalingam you could refer to this link: https://sparkbyexamples.com/spark/spark-sql-join-on-multiple-columns/

Re: How to perform Inner join using withcolumn

shan_chandra — Tue, 31 Jan 2023 15:55:15 GMT

@prudhvi ramalingam - Please refer to the below example code.

import org.apache.spark.sql.functions.expr
val person = Seq(
    (0, "Bill Chambers", 0, Seq(100)),
    (1, "Matei Zaharia", 1, Seq(500, 250, 100)),
    (2, "Michael Armbrust", 1, Seq(250, 100)))
  .toDF("id", "name", "graduate_program", "spark_status")
 
val graduateProgram = Seq(
    (0, "Masters", "School of Information", "UC Berkeley"),
    (2, "Masters", "EECS", "UC Berkeley"),
    (1, "Ph.D.", "EECS", "UC Berkeley"))
  .toDF("id", "degree", "department", "school")
 
val sparkStatus = Seq(
    (500, "Vice President"),
    (250, "PMC Member"),
    (100, "Contributor"))
  .toDF("id", "status")
 
person
  .withColumnRenamed("id", "personId")
  .join(sparkStatus, expr("array_contains(spark_status, id)"))
  .show()