Thanks HubertDudek.

I think using the new library has its own limitations for e.g

i tried doing the predictions based on pandas on spark but its giving error as below though it works fine on normal pandas df.

ValueError: Expected 2D array, got 1D array instead:

data_df=df.to_pandas_on_spark()

#procssed_df is generated after feature engineering on df

inputDf=processed_df.to_pandas_on_spark()

data_df['SCORE']=model.decision_function(inputDf.drop('TEST_VAR4',axis=1))