cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Getting error : Analysis Exception : olumn Is There a PO#17748 are ambiguous. It's probably because you joined several Datasets together, and some of these Datasets are the same. This column points to one of the Datasets but Spark.

vijaykumarbotla
New Contributor III

AnalysisException: Column Is There a PO#17748 are ambiguous. It's probably because you joined several Datasets together, and some of these Datasets are the same. This column points to one of the Datasets but Spark is unable to figure out which one. Please alias the Datasets with different names via `Dataset.as` before joining them, and specify the column using qualified name, e.g. `df.as("a").join(df.as("b"), $"a.id" > $"b.id")`. You can also set spark.sql.analyzer.failAmbiguousSelfJoin to false to disable this check.

Below is the code

reguhjoin = reguhjoin.join(bseg_4j_c2, reguhjoin.conc2 == bseg_4j_c2['Concatenate 2'], how='left').select(reguhjoin["*"], bseg_4j_c2["Is There a PO"])

the above code is working fine in lower environments and we try to migrate to PRD and getting above error, please suggest.

1 ACCEPTED SOLUTION

Accepted Solutions

vijaykumarbotla
New Contributor III

Hi All,

the solution for this problem is very strange.

this has caused due to the version of the Databricks runtime.

We are using Runtime version 7.0 with Apache Spark 3.0.0 version.

In PRD we are using Runtime version 11.3LTS with Apache Spark 3.3.0 version.

When we downgraded our PRD to 7.3 version with Apache Spark 3.0, the issue is resolved.

Why the basic function of Join has changed from one version to another?

View solution in original post

5 REPLIES 5

Debayan
Esteemed Contributor III
Esteemed Contributor III

Hi, Is this the same issue as https://community.databricks.com/s/feed/0D58Y0000AOTu0ySQD? Please tag @Debayan Mukherjee​ , so that I will get notified with your next response.

Yes it is the same issue, I have posted it twice mistakenly.

Lakshay
Esteemed Contributor
Esteemed Contributor

Does the column "Is There a PO" exists in both the dataframes i.e. reguhjoin and bseg_4j_c2?

"Is There a PO" exists in bseg_4j_c2 dataframe, we are trying to include bseg_4j_c2 dataframe into reguhjoin dataframe.

vijaykumarbotla
New Contributor III

Hi All,

the solution for this problem is very strange.

this has caused due to the version of the Databricks runtime.

We are using Runtime version 7.0 with Apache Spark 3.0.0 version.

In PRD we are using Runtime version 11.3LTS with Apache Spark 3.3.0 version.

When we downgraded our PRD to 7.3 version with Apache Spark 3.0, the issue is resolved.

Why the basic function of Join has changed from one version to another?

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!