Copy into checkpoint location not able to find

[HW413]

Hi All,

I have been using COPYINTO for ingesting the data from managed volumes  and my destination is a managed delta table .I would like to know where is it storing the metadata information or a checkpoint location to maintain its idempotent feature.Well i have been pointing to directory not individual files .Please help me to understand about it.I tried everything to search the checkpoint file but am not getting it 

COPY INTO dev.final_test.invoice

FROM "/Volumes/workspace/default/hustest/invoice/"

FILEFORMAT = CSV

FORMAT_OPTIONS ('header' = 'true')

COPY_OPTIONS ('mergeSchema'='true')

Regards,

Husna

[Khaja_Zaffer]

Hello @HW413 

Good day!

Lets say I have a copy into for a table from a volume, once I do this, 

I ran describe table 

I could follow the path associated to this table. if you follow the path: You can check the delta logs for metadata and file size, file modified dates etc., 

[HW413]

@Khaja_Zaffer  my delta table is a managed table i have not given any location path during table creation.

When i run your query i get the below

[Khaja_Zaffer]

Hello @HW413 

That is expected behavior because Unity Catalog fully manages and abstracts the underlying storage for these tables, handling all aspects of read, write, storage, and optimization automatically.

This design ensures centralized governance, security enforcement (e.g., preventing direct file system access that could bypass access controls), and lifecycle management—such as automatic data deletion after DROP TABLE (with a 7-day soft-delete retention for recovery) and built-in optimizations like auto-compaction—without exposing internal paths. 

You can also refer this community document for the same. 

https://community.databricks.com/t5/data-engineering/viewing-managed-delta-table-files/td-p/125741

[szymon_dybczak]

Hi @HW413 ,

You won't find checkpoint. COPY INTO does not use checkpoint like autoloader or spark structured streaming. 

The COPY INTO command retrieves metadata about all files in the specified source directory/prefix . So, every time you run copy into, the command first will create in-memory index of all files. You can see it yourself in SPARK UI:

Then it will compare which files has been already loaded using delta log and this in-memory index of files build in previous step.