Error using COPY INTO after changing schema name

Splush_
New Contributor III

‎07-17-2025 05:51 AM

Hey guys,

I found a weird bug with the COPY INTO command. I have copied a folder in Azure Cloud Storage with a delta table. This worked perfectly. But after changing the name of the schema for this table, it stopped working because it keeps trying to copy the data into the old table which now does not exist anymore.

Splush__0-1752756534234.png

I have tried deleting the Volume, the schema and the table and recreating it without any luck.

Best regards

Samuel

 

0 Kudos

saurabh18cs
Honored Contributor III

‎07-17-2025 08:20 AM

REFRESH TABLE <catalog>.<new_schema>.<table_name>;

MSCK REPAIR TABLE <new_schema>.<table_name>;

Restart Your Cluster

0 Kudos

lingareddy_Alva
Esteemed Contributor

‎07-17-2025 08:34 AM

Hi @Splush_ 

This is a common caching issue in Databricks when working with COPY INTO operations.
The system is holding onto metadata about the old schema location even after you've renamed it.

Clear the COPY INTO operation history:

COPY INTO {new_landing_table_name}
FROM '{file_location}'
FILEFORMAT = {file_type}
FORMAT_OPTIONS ({format_options})
COPY_OPTIONS ("force" = "true", "mergeSchema" = "true")

Root Cause Prevention
The issue occurs because COPY INTO maintains internal metadata about file processing
history tied to the table location. When you rename the schema, this metadata still references the old path.

For future schema changes:
1. Always use DROP TABLE IF EXISTS before recreating
2. Consider using CLONE operations for table migrations
3. Use the force = true option in COPY_OPTIONS to override metadata checks

Try the force = true option first, as it's the most direct solution for this specific caching issue.

LR
0 Kudos