How to translate Apache Pig FOREACH GENERATE statement to Spark?

User15787040559 — Fri, 18 Jun 2021 17:01:08 GMT

If you have the following Apache Pig FOREACH GENERATE statement:

XBCUD_Y_TMP1 = FOREACH (FILTER XBCUD BY act_ind == 'Y') GENERATE cust_hash_key,CONCAT(brd_abbr_cd,ctry_cd) as brdCtry:chararray,updt_dt_hash_key;

the equivalent code in Apache Spark is:

XBCUD_Y_TMP1_DF = (XBCUD_DF
    .filter(col("act_ind") == "Y")
    .select(col("cust_hash_key"),
            concat(col("brd_abbr_cd"),col("ctry_cd")).alias("brdCtry"),
            col("updt_dt_hash_key"))
        )

Re: How to translate Apache Pig FOREACH GENERATE statement to Spark?

User15725630784 — Fri, 18 Jun 2021 19:28:11 GMT

the equivalent code in Apache Spark is:

XBCUD_Y_TMP1_DF = (XBCUD_DF
.filter(col("act_ind") == "Y")
.select(col("cust_hash_key"),
concat(col("brd_abbr_cd"),col("ctry_cd")).alias("brdCtry"),
col("updt_dt_hash_key"))
)

topic Re: How to translate Apache Pig FOREACH GENERATE statement to Spark? in Data Engineering

How to translate Apache Pig FOREACH GENERATE statement to Spark?

Re: How to translate Apache Pig FOREACH GENERATE statement to Spark?