Registering a dataframe coming from a CDC data stream removes the CDC columns from the resulting temporary view, even when explicitly adding a copy of the column to the dataframe.

Louis_Databrick
New Contributor II

‎05-31-2023 12:12 AM 

                            df_source_records.filter(F.col("_change_type").isin("delete", "insert", "update_postimage"))
                            .withColumn("ROW_NUMBER", F.row_number().over(window))
                            .filter("ROW_NUMBER = 1")
                            .withColumn("change_indicator", F.col("_change_type"))
                            .drop("_commit_version", "_commit_timestamp","ROW_NUMBER")
 
 
df_source_records.createOrReplaceGlobalTempView (temporary_view_name)

When now selecting from temporary_view_name, neither the _change_type column nor the change_indicator columns are available to select from. Why is this the case? Seems like a bug? Or is there a workaround?

Thanks!

Labels:
0 Kudos