topic Re: pyspark - execute dynamically framed action statement stored in string variable in Data Engineering

pyspark - execute dynamically framed action statement stored in string variable

dineshg — Thu, 27 Oct 2022 15:08:56 GMT

I need to execute union statement which is framed dynamically and stored in string variable. I framed the union statement, but struck with executing the statement. Does anyone know how to execute union statement stored in string variable? I'm using pyspark in databricks notebook.

df1 = df.filter((col("vchDataSection") == "AccountMasterInfo") & (col("bActive") == 1)).withColumn("dfs", concat(lit(".union(df"), col("iRuleid"), lit(")")))

df2 = df1.agg(concat_ws("",collect_list(col("dfs")))).withColumnRenamed("concat_ws(, collect_list(dfs))", "AccInfoRules").withColumn("replacestr",lit(""))

df3 = df2.select(overlay("AccInfoRules","replacestr",1,7).alias("overlayed"))

var_a = df3.collect()

var_a = var_a[0].__getitem__('overlayed')

var_b = var_a.replace(')', '', 1)

print(var_b)

o/p: df533.union(df534).union(df535).union(df536)

Re: pyspark - execute dynamically framed action statement stored in string variable

Shalabh007 — Tue, 29 Nov 2022 10:18:49 GMT

@Dineshkumar Gopalakrishnan using python's exec() function can be used to execute a python statement, which in your case could be pyspark union statement. Refer below sample code snippet for your reference.

df1 = spark.sparkContext.parallelize([(1, 2, ["1", "2", "3"]), (1, 3, ["4", "1", "5", "6"]) , (2, 4, ["2"]),(2, 5, ["3"])]).toDF(["store", "count", "values"])
 
df2 = spark.sparkContext.parallelize([(3, 2, ["1", "2", "3"]), (3, 3, ["4", "1", "5", "6"]) , (4, 4, ["2"]),(4, 5, ["3"])]).toDF(["store", "count", "values"])
 
union_statment = "df = df1.union(df2)"
exec(union_statment)

Above code will execute the pyspark union api on df1 and df2 and will assign the result to dataframe 'df'.

You can have more complex union statment as part of your dynamic string

Re: pyspark - execute dynamically framed action statement stored in string variable

dineshg — Thu, 01 Dec 2022 15:40:52 GMT

Thank you @Shalabh Agarwal above solution worked for me.

Re: pyspark - execute dynamically framed action statement stored in string variable

Shalabh007 — Thu, 01 Dec 2022 15:46:59 GMT

amazing @Dineshkumar Gopalakrishnan

Can you please click on the "Select As Best" button if you find the information provided helps resolve your question.