topic Re: How is Idempotency ensured for COPY INTO command in Machine Learning

How is Idempotency ensured for COPY INTO command

brickster_2018 — Fri, 25 Jun 2021 14:01:28 GMT

Re: How is Idempotency ensured for COPY INTO command

brickster_2018 — Fri, 25 Jun 2021 14:02:02 GMT

COPY INTO command internally uses key-value store - RocksDB to store the details of the input files. This information is stored inside the Delta table log directory. This acts like the checkpointing information for a streaming query. Next time a COPY INTO command is triggered on the same table, as a first step, the data from the RocksDB is loaded and compared against the input files. Under the hood, a dedupe logic is performed to ensure idempotency.

More details here:

https://docs.databricks.com/spark/latest/spark-sql/language-manual/delta-copy-into.html

For COPY_OPTIONS, the parameter force if set to 'true', idempotency is disabled and files are loaded regardless of whether they’ve been loaded before.

Re: How is Idempotency ensured for COPY INTO command

N_M — Fri, 01 Dec 2023 15:27:10 GMT

How does COPY_INTO work with table restore?

I made some tests, and the restore method does NOT restore the key-store values of the target at the specific version, which means that the data that came after the chosen version cannot be inserted (unless forced).

Is this behavior intended?