cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Error using COPY INTO after changing schema name

Splush_
New Contributor III

Hey guys,

I found a weird bug with the COPY INTO command. I have copied a folder in Azure Cloud Storage with a delta table. This worked perfectly. But after changing the name of the schema for this table, it stopped working because it keeps trying to copy the data into the old table which now does not exist anymore.

Splush__0-1752756534234.png

I have tried deleting the Volume, the schema and the table and recreating it without any luck.

Best regards

Samuel

 

2 REPLIES 2

saurabh18cs
Honored Contributor

REFRESH TABLE <catalog>.<new_schema>.<table_name>;

MSCK REPAIR TABLE <new_schema>.<table_name>;

Restart Your Cluster

lingareddy_Alva
Honored Contributor III

Hi @Splush_ 

This is a common caching issue in Databricks when working with COPY INTO operations.
The system is holding onto metadata about the old schema location even after you've renamed it.

Clear the COPY INTO operation history:

COPY INTO {new_landing_table_name}
FROM '{file_location}'
FILEFORMAT = {file_type}
FORMAT_OPTIONS ({format_options})
COPY_OPTIONS ("force" = "true", "mergeSchema" = "true")

Root Cause Prevention
The issue occurs because COPY INTO maintains internal metadata about file processing
history tied to the table location. When you rename the schema, this metadata still references the old path.

For future schema changes:
1. Always use DROP TABLE IF EXISTS before recreating
2. Consider using CLONE operations for table migrations
3. Use the force = true option in COPY_OPTIONS to override metadata checks

Try the force = true option first, as it's the most direct solution for this specific caching issue.

LR