topic Getting error : Analysis Exception : olumn Is There a PO#17748 are ambiguous. It's probably because you joined several Datasets together, and some of these Datasets are the same. This column points to one of the Datasets but Spark. in Data Engineering

Getting error : Analysis Exception : olumn Is There a PO#17748 are ambiguous. It's probably because you joined several Datasets together, and some of these Datasets are the same. This column points to one of the Datasets but Spark.

vijaykumarbotla — Mon, 29 May 2023 16:15:23 GMT

AnalysisException: Column Is There a PO#17748 are ambiguous. It's probably because you joined several Datasets together, and some of these Datasets are the same. This column points to one of the Datasets but Spark is unable to figure out which one. Please alias the Datasets with different names via `Dataset.as` before joining them, and specify the column using qualified name, e.g. `df.as("a").join(df.as("b"), $"a.id" > $"b.id")`. You can also set spark.sql.analyzer.failAmbiguousSelfJoin to false to disable this check.

Below is the code

reguhjoin = reguhjoin.join(bseg_4j_c2, reguhjoin.conc2 == bseg_4j_c2['Concatenate 2'], how='left').select(reguhjoin["*"], bseg_4j_c2["Is There a PO"])

the above code is working fine in lower environments and we try to migrate to PRD and getting above error, please suggest.

Re: Getting error : Analysis Exception : olumn Is There a PO#17748 are ambiguous. It's probably because you joined several Datasets together, and some of these Datasets are the same. This column points to one of the Datasets but Spark.

vijaykumarbotla — Tue, 30 May 2023 10:04:31 GMT

Yes it is the same issue, I have posted it twice mistakenly.

Re: Getting error : Analysis Exception : olumn Is There a PO#17748 are ambiguous. It's probably because you joined several Datasets together, and some of these Datasets are the same. This column points to one of the Datasets but Spark.

Lakshay — Tue, 30 May 2023 11:04:24 GMT

Does the column "Is There a PO" exists in both the dataframes i.e. reguhjoin and bseg_4j_c2?

Re: Getting error : Analysis Exception : olumn Is There a PO#17748 are ambiguous. It's probably because you joined several Datasets together, and some of these Datasets are the same. This column points to one of the Datasets but Spark.

vijaykumarbotla — Wed, 31 May 2023 00:56:34 GMT

"Is There a PO" exists in bseg_4j_c2 dataframe, we are trying to include bseg_4j_c2 dataframe into reguhjoin dataframe.

Re: Getting error : Analysis Exception : olumn Is There a PO#17748 are ambiguous. It's probably because you joined several Datasets together, and some of these Datasets are the same. This column points to one of the Datasets but Spark.

vijaykumarbotla — Wed, 31 May 2023 13:56:24 GMT

Hi All,

the solution for this problem is very strange.

this has caused due to the version of the Databricks runtime.

We are using Runtime version 7.0 with Apache Spark 3.0.0 version.

In PRD we are using Runtime version 11.3LTS with Apache Spark 3.3.0 version.

When we downgraded our PRD to 7.3 version with Apache Spark 3.0, the issue is resolved.

Why the basic function of Join has changed from one version to another?

Re: Getting error : Analysis Exception : olumn Is There a PO#17748 are ambiguous. It's probably because you joined several Datasets together, and some of these Datasets are the same. This column points to one of the Datasets but Spark.

Debayan — Tue, 30 May 2023 06:59:10 GMT

Hi, Is this the same issue as https://community.databricks.com/s/feed/0D58Y0000AOTu0ySQD? Please tag @Debayan Mukherjee , so that I will get notified with your next response.