To to convert pd df to a dict before passing on E.g. createDataFrame(pd.read_csv('abc').to.dict())In my case there was some non serializable attribute on pd df which is dropped when converting to dict. Btw logging in here and answering was an insane ...
