- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-27-2022 08:08 AM
I need to execute union statement which is framed dynamically and stored in string variable. I framed the union statement, but struck with executing the statement. Does anyone know how to execute union statement stored in string variable? I'm using pyspark in databricks notebook.
df1 = df.filter((col("vchDataSection") == "AccountMasterInfo") & (col("bActive") == 1)).withColumn("dfs", concat(lit(".union(df"), col("iRuleid"), lit(")")))
df2 = df1.agg(concat_ws("",collect_list(col("dfs")))).withColumnRenamed("concat_ws(, collect_list(dfs))", "AccInfoRules").withColumn("replacestr",lit(""))
df3 = df2.select(overlay("AccInfoRules","replacestr",1,7).alias("overlayed"))
var_a = df3.collect()
var_a = var_a[0].__getitem__('overlayed')
var_b = var_a.replace(')', '', 1)
print(var_b)
o/p: df533.union(df534).union(df535).union(df536)
- Labels:
-
Databricks notebook
-
Pyspark