05-29-2023 09:15 AM
AnalysisException: Column Is There a PO#17748 are ambiguous. It's probably because you joined several Datasets together, and some of these Datasets are the same. This column points to one of the Datasets but Spark is unable to figure out which one. Please alias the Datasets with different names via `Dataset.as` before joining them, and specify the column using qualified name, e.g. `df.as("a").join(df.as("b"), $"a.id" > $"b.id")`. You can also set spark.sql.analyzer.failAmbiguousSelfJoin to false to disable this check.
Below is the code
reguhjoin = reguhjoin.join(bseg_4j_c2, reguhjoin.conc2 == bseg_4j_c2['Concatenate 2'], how='left').select(reguhjoin["*"], bseg_4j_c2["Is There a PO"])
the above code is working fine in lower environments and we try to migrate to PRD and getting above error, please suggest.
05-31-2023 06:56 AM
Hi All,
the solution for this problem is very strange.
this has caused due to the version of the Databricks runtime.
We are using Runtime version 7.0 with Apache Spark 3.0.0 version.
In PRD we are using Runtime version 11.3LTS with Apache Spark 3.3.0 version.
When we downgraded our PRD to 7.3 version with Apache Spark 3.0, the issue is resolved.
Why the basic function of Join has changed from one version to another?
05-29-2023 11:59 PM
Hi, Is this the same issue as https://community.databricks.com/s/feed/0D58Y0000AOTu0ySQD? Please tag @Debayan Mukherjee , so that I will get notified with your next response.
05-30-2023 03:04 AM
Yes it is the same issue, I have posted it twice mistakenly.
05-30-2023 04:04 AM
Does the column "Is There a PO" exists in both the dataframes i.e. reguhjoin and bseg_4j_c2?
05-30-2023 05:56 PM
"Is There a PO" exists in bseg_4j_c2 dataframe, we are trying to include bseg_4j_c2 dataframe into reguhjoin dataframe.
05-31-2023 06:56 AM
Hi All,
the solution for this problem is very strange.
this has caused due to the version of the Databricks runtime.
We are using Runtime version 7.0 with Apache Spark 3.0.0 version.
In PRD we are using Runtime version 11.3LTS with Apache Spark 3.3.0 version.
When we downgraded our PRD to 7.3 version with Apache Spark 3.0, the issue is resolved.
Why the basic function of Join has changed from one version to another?
Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections.
Click here to register and join today!
Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.