Often times just writing a question helps resolve it. For anyone facing issues with databricks-connect that don't show up using databricks directly, here is a list of limitations (rtfm to me):

https://learn.microsoft.com/en-us/azure/databricks/dev-tools/databricks-connect#limitations

And in particular watch for the size of the dataframe. databricks-connect doesn't support dataframes larger than 128 MB, which is not much. Hopefully next releases will allow larger dataframes.

Hope this helps!