lmu
Databricks Partner

After further testing, it was found that the dedicated access mode (formerly single user) either does not work or exhibits strange behaviour. In one scenario, the 16.4 cluster with dedicated access mode could write in append mode but not overwrite, and upsert led to nulls in the test table. This was very strange.
However, it worked with the shared access mode.

Working solution:

  • 15.4 dedicated cluster with standard access mode (formerly shared)
  • 16.4 dedicated cluster with standard access mode (formerly shared)

Not working:

  • 15.4 with access mode dedicated(single user) cluster
  • 16.4 withaccess mode dedicated(single user) cluster
  • 13.4 dedicated or shared

View solution in original post