I've been encountering a similar issue when upgrading to 12.2, and started stripping back all the changes, to the point where I'm basically running the same exact code in 12.2 and 11.3 and it works in 11.3 but gives a similar error to the original question.

I stripped it back even further to just creating a test that only deals with trying to do a DataTable merge and am able to simulate it. Not sure it would match the situation above perfectly, but it might help with the reproduction of it.

Basically, I've got two different clusters, one running each version that I can switch between, and I've got a scala notebook (attached), it contains a Sql script at the top that generates the Delta Table to write to (that simulates the schema we're using), and then code to create a simple data frame to merge into it.

As I say it seems to run fine in 11.3, but does not in 12.2. Any thoughts on if there is an issue with the merge approach, or ways around it? We use it on a significant portion of our tasks and it prevents us from updating to 12.2, which we're hoping to do for some other things we would gain from it?

Thanks, Tom