topic Best ways to copy the parquet files in the staging directory to the Delta table in Data Engineering https://community.databricks.com/t5/data-engineering/best-ways-to-copy-the-parquet-files-in-the-staging-directory-to/m-p/19298#M12920 <P>I have some parquet data in a temporary directory. Can I copy them into the delta table directly, what are the best options. </P> Fri, 25 Jun 2021 17:32:33 GMT brickster_2018 2021-06-25T17:32:33Z Best ways to copy the parquet files in the staging directory to the Delta table https://community.databricks.com/t5/data-engineering/best-ways-to-copy-the-parquet-files-in-the-staging-directory-to/m-p/19298#M12920 <P>I have some parquet data in a temporary directory. Can I copy them into the delta table directly, what are the best options. </P> Fri, 25 Jun 2021 17:32:33 GMT https://community.databricks.com/t5/data-engineering/best-ways-to-copy-the-parquet-files-in-the-staging-directory-to/m-p/19298#M12920 brickster_2018 2021-06-25T17:32:33Z Re: Best ways to copy the parquet files in the staging directory to the Delta table https://community.databricks.com/t5/data-engineering/best-ways-to-copy-the-parquet-files-in-the-staging-directory-to/m-p/19299#M12921 <P>The easiest solution is to use the COPY INTO command. The COPY INTO command ensures idempotency, so even if the operation fails there are no data inconsistencies. COPY INTO command utilizes the resources on the Spark cluster hence completes faster.&nbsp;</P><P></P><P><A href="https://docs.databricks.com/spark/latest/spark-sql/language-manual/delta-copy-into.html" target="test_blank">https://docs.databricks.com/spark/latest/spark-sql/language-manual/delta-copy-into.html</A></P> Fri, 25 Jun 2021 17:33:01 GMT https://community.databricks.com/t5/data-engineering/best-ways-to-copy-the-parquet-files-in-the-staging-directory-to/m-p/19299#M12921 brickster_2018 2021-06-25T17:33:01Z