I have the following sparkdataframe :
agent_id/ payment_amount
a /1000
b /1100
a /1100
a /1200
b /1200
b /1250
a /10000
b /9000
my desire output would be something like
<code>agen_id 95_quantile
a whatever is95 quantile for agent a payments
b whatever is95 quantile for agent b payments
for each group of agent_id i need to calculate the 0.95 quantile, i take the following approach:
<code>test_df.groupby('agent_id').approxQuantile('payment_amount',0.95)
but i take the following error:
<code>'GroupedData' object has no attribute 'approxQuantile'
i need to have .95 quantile(percentile) in a new column so later can be used for filtering purposes