topic Re: calculate median and inter quartile range on spark dataframe in Data Engineering

calculate median and inter quartile range on spark dataframe

NarwshKumar — Sat, 06 Feb 2016 15:11:12 GMT

I have a spark dataframe of 5 columns and I want to calculate median and interquartile range on all. I am not able to figure out how do I write udf and call them on columns.

Re: calculate median and inter quartile range on spark dataframe

raela — Tue, 09 Feb 2016 00:23:01 GMT

Hi,

You can refer to Databricks' docs on how to create UDFs.

Python UDFs | Scala UDFs

Re: calculate median and inter quartile range on spark dataframe

rlgarris — Tue, 09 Feb 2016 15:42:10 GMT

Hi,

You can also using Windowing functions https://databricks.com/blog/2015/07/15/introducing-window-functions-in-spark-sql.html';

percent_rank at 0.25, 0.50 and 0.75 will give you want you are looking for.

Re: calculate median and inter quartile range on spark dataframe

jmwilli25 — Tue, 23 May 2017 22:28:53 GMT

Here is the easiest way to calculate this... https://stackoverflow.com/questions/37032689/scala-first-quartile-third-quartile-and-iqr-from-spark-sqlcontext-dataframe

No Hive or windowing necessary.