Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-07-2019 03:30 PM
@Mike K. I have added a solution without using views.
temp_df = df2.join(df1, "colA", how = "left").select("*", df1.colB.alias("df1_colB")).drop(df1.colB)df2 = temp_df.withColumn("colB", when((col("colB") <= col("df1_colB")) \
& (temp_df.df1_colB.isNotNull()) \
& (temp_df.colC == 'XYZ'), col("colB") + col("df1_colB")).otherwise(col("colB"))) \
.drop("df1_colB")Link to the full code in notebook w/ explanation:
https://github.com/bsthapit/Databricks_Solutions/blob/master/db_29380.ipynb