Databricks Community

Johan_Van_Noten · 12-06-2021

EnvironmentAzure Databricks 10.1, including Spark 3.2.0ScenarioI want to retrieve the average of a series of values between two timestamps, using a SQL UDF.The average is obviously just an example. In a real scenario, I would like to hide some additi...

Johan_Van_Noten · 12-02-2022

Thanks guys, @Lav Chandaria and @Raghu Bindinganavale 's solutions both work, but as indicated in my reply above, I'm worried about the performance of evaluating the datediff (Lav) or the label (Raghu) over the full dataset without the engine being...

Johan_Van_Noten · 12-02-2022

Hi @Raghu Bindinganavale ,Thanks for your suggestion.I'm not sure how the SQL engine will optimize this query, but you are essentially calculating the condition-label over all oventemperatures in the full storage (millions) and only afterwards filte...

Johan_Van_Noten · 12-02-2022

Hi @Raghu Bindinganavale ,I wasn't able to work around this and considered it a restriction of SparkSQL. We are coping with it on a higher layer now, which is unfortunate.Thanks,Johan

Johan_Van_Noten · 07-05-2022

Hi Robert,Thanks for your suggestion. The fact that I want to do this in SparkSQL is because there is no underlying SQLServer. Well, there might be, but often there isn't. We make use of the possibility of SparkSQL to abstract the underlying data sto...

Johan_Van_Noten · 07-05-2022

Seems to be a duplicate of your comment on https://community.databricks.com/s/question/0D53f00001XCuCACA1/correlated-column-exception-in-sql-udf-when-using-udf-parameters. I guess you did that to be able to put other tags?

Databricks Community

User Stats

User Activity

Correlated column exception in SQL UDF when using UDF parameters.

Re: Correlated column exception in SQL UDF when using UDF parameters.

Re: Correlated column exception in SQL UDF when using UDF parameters.

Re: Correlated column exception in SQL UDF when using UDF parameters.

Re: Correlated column exception in SQL UDF when using UDF parameters.

Re: Correlated Column Exception in Spark SQL