Cannot resolve due to data type mismatch: incompatible types ("STRING" and ARRAY

oussValrho — Thu, 22 Feb 2024 15:01:25 GMT

hey i have this error from a while :
Cannot resolve "(needed_skill_id = needed_skill_id)" due to data type mismatch: the left and right operands of the binary operator have incompatible types ("STRING" and "ARRAY<STRING>"). SQLSTATE: 42K09;
and these are the lines responsible for that :

to_update = current_link_df.where(col("active") == True).join(dataframe, ["position_id", "needed_skill_id"], how="left_anti")

(

DeltaTable.forName(spark, table_name).alias("t")

.merge(to_update.withColumn("end_date", current_date()).alias("s"), "s.position_id = t.position_id AND s.needed_skill_id = t.needed_skill_id")

Any idea ?

Thank you !!

topic Cannot resolve due to data type mismatch: incompatible types ("STRING" and ARRAY<STRING> in Data Engineering

Cannot resolve due to data type mismatch: incompatible types ("STRING" and ARRAY