Hi, 

According to When to partition tables on Databricks :

• Databricks recommends you do not partition tables that contain less than a terabyte of data.
• If you proceed with partitions, please check if all partitions contain at least a gigabyte of data. 

Instead of partitions, take a look at :

• Managing files and indexing data with Delta Lake :  OPTIMIZE, ZORDER, ANALYZE and VACUUM are the commands that might be super useful while you need to enhance the processing performance
• Predictive optimization for Delta Lake : executes OPTIMIZE and VACUUM automatically for you (Unity Catalog + managed tables are the prerequisites for this feature as of now)

It is highly possible you do not need to rewrite the whole dataset but rather use the MERGE operation.

While running the processes that may work with the same partitions please make the separation as explicit as possible in the operation condition (see ConcurrentAppendException).

You can find here a repository with demos that can contain useful hints and that you can install in your workspace (maybe the one on Delta Lake would be the most relevant for you at the current stage ? click on the "View the Notebooks" button to access to codes and run the pip command to play with the content)

Hope it helps,

Best,