The versions are compatible 15.4 LTS ML uses py 3.11 and spark 3.5.0

I am not using a docker image

shouldn't 32G and 16cores be more than enough for 200k-1m rows? They dont contain anything more than doubles and short strings and with only 20 columns they shouldn't be "large"

"This could be due to network issues, resource constraints, or other transient failures." - if there are network issues, or transient failures how do I go about resolving them. 

Finally, is using the FeatureEngineeringClient the best way to go about writing to tables? Could I use df.write.saveAsTable()?