I use the following option to write from multiple tasks to the same table with overwrite (in Pyspark)
.option("partitionOverwriteMode", "dynamic")
The table was created with partition by so it works as expected.
I read about liquid clustering and it's benefits over table partitioning
Is it possible to perform the same use case of writing from multiple tasks (simultaneously) to the same table with overwrite on a liquid clustered table?
Thanks