topic Re: Error using COPY INTO after changing schema name in Data Engineering

Error using COPY INTO after changing schema name

Splush_ — Thu, 17 Jul 2025 12:51:10 GMT

Hey guys,

I found a weird bug with the COPY INTO command. I have copied a folder in Azure Cloud Storage with a delta table. This worked perfectly. But after changing the name of the schema for this table, it stopped working because it keeps trying to copy the data into the old table which now does not exist anymore.

I have tried deleting the Volume, the schema and the table and recreating it without any luck.

Best regards

Samuel

Re: Error using COPY INTO after changing schema name

saurabh18cs — Thu, 17 Jul 2025 15:20:21 GMT

REFRESH TABLE <catalog>.<new_schema>.<table_name>;

MSCK REPAIR TABLE <new_schema>.<table_name>;

Restart Your Cluster

Re: Error using COPY INTO after changing schema name

lingareddy_Alva — Thu, 17 Jul 2025 15:34:45 GMT

Hi @Splush_

This is a common caching issue in Databricks when working with COPY INTO operations.
The system is holding onto metadata about the old schema location even after you've renamed it.

Clear the COPY INTO operation history:

COPY INTO {new_landing_table_name}
FROM '{file_location}'
FILEFORMAT = {file_type}
FORMAT_OPTIONS ({format_options})
COPY_OPTIONS ("force" = "true", "mergeSchema" = "true")

Root Cause Prevention
The issue occurs because COPY INTO maintains internal metadata about file processing
history tied to the table location. When you rename the schema, this metadata still references the old path.

For future schema changes:
1. Always use DROP TABLE IF EXISTS before recreating
2. Consider using CLONE operations for table migrations
3. Use the force = true option in COPY_OPTIONS to override metadata checks

Try the force = true option first, as it's the most direct solution for this specific caching issue.