Best ways to copy the parquet files in the staging directory to the Delta table

[User16869510359]

I have some parquet data in a temporary directory. Can I copy them into the delta table directly, what are the best options.

[User16869510359 · Accepted solution]

The easiest solution is to use the COPY INTO command. The COPY INTO command ensures idempotency, so even if the operation fails there are no data inconsistencies. COPY INTO command utilizes the resources on the Spark cluster hence completes faster. 

https://docs.databricks.com/spark/latest/spark-sql/language-manual/delta-copy-into.html