I did nothing different than just switch the Runtime from 15.4 ML LTS to 16.1 ML and it write to the table just fine. I dont know exactly why it worked but I believe that the spark version might be different between the two and that could be the cause. The exact reason is still ambiguous to me.