Getting error : Analysis Exception : olumn Is There a PO#17748 are ambiguous. It's probably because you joined several Datasets together, and some of these Datasets are the same. This column points to one of the Datasets but Spark.

[vijaykumarbotla]

AnalysisException: Column Is There a PO#17748 are ambiguous. It's probably because you joined several Datasets together, and some of these Datasets are the same. This column points to one of the Datasets but Spark is unable to figure out which one. Please alias the Datasets with different names via `Dataset.as` before joining them, and specify the column using qualified name, e.g. `df.as("a").join(df.as("b"), $"a.id" > $"b.id")`. You can also set spark.sql.analyzer.failAmbiguousSelfJoin to false to disable this check.

Below is the code

reguhjoin = reguhjoin.join(bseg_4j_c2, reguhjoin.conc2 == bseg_4j_c2['Concatenate 2'], how='left').select(reguhjoin["*"], bseg_4j_c2["Is There a PO"])

the above code is working fine in lower environments and we try to migrate to PRD and getting above error, please suggest.

[Debayan]

Hi, Is this the same issue as https://community.databricks.com/s/feed/0D58Y0000AOTu0ySQD? Please tag @Debayan Mukherjee​ , so that I will get notified with your next response.

[vijaykumarbotla]

Yes it is the same issue, I have posted it twice mistakenly.

[Lakshay]

Does the column "Is There a PO" exists in both the dataframes i.e. reguhjoin and bseg_4j_c2?

[vijaykumarbotla]

"Is There a PO" exists in bseg_4j_c2 dataframe, we are trying to include bseg_4j_c2 dataframe into reguhjoin dataframe.

[vijaykumarbotla · Accepted solution]

Hi All,

the solution for this problem is very strange.

this has caused due to the version of the Databricks runtime.

We are using Runtime version 7.0 with Apache Spark 3.0.0 version.

In PRD we are using Runtime version 11.3LTS with Apache Spark 3.3.0 version.

When we downgraded our PRD to 7.3 version with Apache Spark 3.0, the issue is resolved.

Why the basic function of Join has changed from one version to another?