Yes, I think the best will be to rebuild the code entirely and use, for example, COPY INTO.

• dbutils utilize just one core
• rdd, is not optimized by the catalyst, and AQE
• high-level code like COPY INTO is executed distributed way and is optimized