Hi!

I am inserting a pyspark dataframe to Azure sql server and it takes a very long time. The database is a s4 but my dataframe that is 17 million rows and 30 columns takes up to 50 minutes to insert.

Is there a way to significantly speed this up? I am open for many things - switching connector, run the job in scala instead and so on. I assume my dataframe is not even that big so there must be some way to speed this up.

Also, when the dataframe is of this size, the target table is usually empty.

Thanks in advance!