cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

copy into - statement executed - no records inserted

DoredlaCharan
New Contributor III

I am able to run the copy command but records are not inserting and there is no error raised on execution.

1 ACCEPTED SOLUTION

Accepted Solutions

szymon_dybczak
Esteemed Contributor III

Hi @DoredlaCharan ,

Maybe that file was already read? To check if that's the case you can first try to use force option.
Add COPY_OPTIONS('force' = 'true') to your code.

COPY INTO nxt_dev.raw_vam_viraf.fact_servicer_report
FROM (
  SELECT
    ...
  FROM 's3://snowbergsandbox/raw_vam_viraf/fact_servicer_report/VAM VIRAF PTC_Servicer Report_20260216_167309_387664.csv'
)
FILEFORMAT = CSV
FORMAT_OPTIONS ('header' = 'true')
COPY_OPTIONS ('force' = 'true');

 Another thing that is worth checking - spaces in your S3 path. The filename has spaces (VAM VIRAF PTC_Servicer Report...).
Try to get rid of that spaces from filename of try URL-encoding them VAM%20VIRAF%20PTC_Servicer%20Report...) 

View solution in original post

7 REPLIES 7

Pat
Esteemed Contributor

Hi @DoredlaCharan ,
it would be nice to see more details. The COPY INTO command you are running (with params). It would be nice to see a screenshot after you haveexecuted the load.

Few things on top of my head:
Idempotency skips files: Previously loaded files (same name) are ignored to ensure exactly-once loading.
Schema mismatches: Data types don't align, or headers/inference options fail silently
Empty or filtered source: No matching files via pattern/FILES, or data filtered out by options.

AnthonyAnand
Databricks Partner

Hi @DoredlaCharan,

Kindly add some more details like what are you trying to do and if possible some screenshot.

Additionally Please try to below steps:

1. REFRESH TABLE table_name and see if this helps in case of metadata sync issue.

2. Delta Log - Check the delta log whether the insert operation happened or you could see the inserted records details ?

 

balajij8
Contributor

Verify if new files are posted before running copy

DoredlaCharan
New Contributor III

DoredlaCharan_0-1771261098958.png

the above is the screenshot, 
Run1 - s3 path - no records ingested
Run2 - s3 path - no records ingested
but no error

Also one more is i tried using another s3 path as well still unable to insert.

DoredlaCharan
New Contributor III

DoredlaCharan_0-1771261098958.png

the above is the screenshot, 
Run1 - s3 path - no records ingested
Run2 - s3 path - no records ingested
but no error, Also one more is i tried using another s3 path as well still unable to insert.

szymon_dybczak
Esteemed Contributor III

Hi @DoredlaCharan ,

Maybe that file was already read? To check if that's the case you can first try to use force option.
Add COPY_OPTIONS('force' = 'true') to your code.

COPY INTO nxt_dev.raw_vam_viraf.fact_servicer_report
FROM (
  SELECT
    ...
  FROM 's3://snowbergsandbox/raw_vam_viraf/fact_servicer_report/VAM VIRAF PTC_Servicer Report_20260216_167309_387664.csv'
)
FILEFORMAT = CSV
FORMAT_OPTIONS ('header' = 'true')
COPY_OPTIONS ('force' = 'true');

 Another thing that is worth checking - spaces in your S3 path. The filename has spaces (VAM VIRAF PTC_Servicer Report...).
Try to get rid of that spaces from filename of try URL-encoding them VAM%20VIRAF%20PTC_Servicer%20Report...) 

shazi
New Contributor III

Records not inserted with the COPY INTO statement usually indicate either no matching source data or unmet conditions in the query.