Databricks Community

Johan_Van_Noten

As one of the steps in my data engineering pipeline, I need to perform a POST request to a http (not -s) server.This all works fine, except for the situation described below: it then hangs indefinitely.Environment:Azure Databricks Runtime 13.3 LTSPyt...

Johan_Van_Noten · 12-06-2021

EnvironmentAzure Databricks 10.1, including Spark 3.2.0ScenarioI want to retrieve the average of a series of values between two timestamps, using a SQL UDF.The average is obviously just an example. In a real scenario, I would like to hide some additi...

Johan_Van_Noten

Thanks for your quick and extensive reply.Given that I don't have any administration rights on the Azure/Databricks environment and don't have the REST-server under control, some of the sensible suggestions are difficult.I will work with IT to check ...

Johan_Van_Noten · 12-02-2022

Thanks guys, @Lav Chandaria and @Raghu Bindinganavale 's solutions both work, but as indicated in my reply above, I'm worried about the performance of evaluating the datediff (Lav) or the label (Raghu) over the full dataset without the engine being...

Johan_Van_Noten · 12-02-2022

Hi @Raghu Bindinganavale ,Thanks for your suggestion.I'm not sure how the SQL engine will optimize this query, but you are essentially calculating the condition-label over all oventemperatures in the full storage (millions) and only afterwards filte...

Johan_Van_Noten · 12-02-2022

Hi @Raghu Bindinganavale ,I wasn't able to work around this and considered it a restriction of SparkSQL. We are coping with it on a higher layer now, which is unfortunate.Thanks,Johan

Johan_Van_Noten · 07-05-2022

Hi Robert,Thanks for your suggestion. The fact that I want to do this in SparkSQL is because there is no underlying SQLServer. Well, there might be, but often there isn't. We make use of the possibility of SparkSQL to abstract the underlying data sto...

Databricks Community

User Stats

User Activity

Long-running Python http POST hangs

Correlated column exception in SQL UDF when using UDF parameters.

Re: Long-running Python http POST hangs

Re: Correlated column exception in SQL UDF when using UDF parameters.

Re: Correlated column exception in SQL UDF when using UDF parameters.

Re: Correlated column exception in SQL UDF when using UDF parameters.

Re: Correlated column exception in SQL UDF when using UDF parameters.