This was recommended on StackOverflow though I haven't tested with ADLS yet.

sc._jsc.hadoopConfiguration().set("mapreduce.fileoutputcommitter.marksuccessfuljobs", "false")

Note it may impact the whole cluster.

You could also use dbutils.fs.rm step to remove any created files.

cheers,

Andrew