Does databricks support spearman rank correlation

[chand20025]

  while working on spark scala. I tried working on code using pearson correaltion it works but when I  tried to work on spearman rank correlation it displayed error like 

Currently only the calculation of the Pearson Correlation coefficient is supported.

[Kaniz_Fatma]

Hi @chand20025, 

It seems that Databricks currently only supports the calculation of the Pearson Correlation coefficient and not the Spearman rank correlation. If you’re specifically looking to calculate Spearman rank correlation, you might need to explore alternative approaches or libraries.

One possible workaround is to use a common table expression (CTE) to compute the rank and then apply the corr() function on the result. Here’s an example in SQL:

WITH ranked_data AS (
    SELECT
        col1,
        col2,
        RANK() OVER (ORDER BY col1) AS rank_col1,
        RANK() OVER (ORDER BY col2) AS rank_col2
    FROM your_table
)
SELECT corr(rank_col1, rank_col2) AS spearman_correlation
FROM ranked_data;

This approach calculates the Spearman rank correlation by first ranking the data and then applying the correlation function. 

If you encounter any further issues or need additional assistance, feel free to ask! 😊