Well my bad, I thought this was the issue, but eventually I reduced the number of rows so that the size became less than 1 MB, and it still failed with the same error, so I still don't know why this fails with databricks-connect, and I have checked that all used spark functions support spark connect. 

So if anyone has any idea, thanks for sharing.