CONVERT TO DELTA fails to merge file schema

deployment_fail — Wed, 04 Feb 2026 16:50:01 GMT

I have a directory of Parquet files in Azure Data Lake Storage that I want to convert to a Delta Lake table. I run this:

CONVERT TO DELTA parquet.`abfss://container@storage_account.dfs.core.windows.net/directory_name`;

But it throws this error: "SparkException: [DELTA_FAILED_MERGE_SCHEMA_FILE] Failed to merge schema of file abfss://container@storage_account.dfs.core.windows.net/directory_name/file_name_123.parquet: ..."

I ran this in an all-purpose cluster with the `spark.databricks.delta.mergeSchema.enabled` config set to true.

Re: CONVERT TO DELTA fails to merge file schema

saurabh18cs — Thu, 05 Feb 2026 10:38:16 GMT

Hi I believe the spark configuration you are trying to use is used when writing to existing delta table and not while converting non-delta table to delta table.with error it is quite evident that your parquet files does not conform to unified scheme hence conversion fails.

lets try to skip automatic scheam inference by suplying the schema like this:

schema = "col1 INT, col2 STRING, col3 DOUBLE"

df = spark.read.schema(schema).parquet("

abfss://container@storage_account.dfs.core.windows.net/directory_name")

and then write to delta:

df.write.format("delta").mode("overwrite").save("abfss://.../delta_table")

share your outcome. thanks

@deployment_fail

Re: CONVERT TO DELTA fails to merge file schema

SteveOstrowski — Sun, 08 Mar 2026 02:54:14 GMT

Hi @deployment_fail,

Good timing on this question. Let me explain what is happening and walk you through several approaches to resolve it.

WHAT IS HAPPENING

When you run CONVERT TO DELTA, Databricks reads the Parquet footer metadata from every file in the directory and attempts to merge all of those individual file schemas into a single unified Delta table schema. The DELTA_FAILED_MERGE_SCHEMA_FILE error means that at least one file has a schema that cannot be reconciled with the others -- for example, the same column name appears with different data types across files.

An important clarification: the spark.databricks.delta.mergeSchema.enabled configuration you set does not apply to the CONVERT TO DELTA command. That setting controls schema evolution for write operations (INSERT, MERGE, DataFrame writes) against existing Delta tables. CONVERT TO DELTA uses its own internal schema merge logic when reading Parquet file footers, so that config has no effect here.

COMMON CAUSES OF THIS ERROR

1. Schema drift over time -- an upstream system changed a column type (e.g., INT to LONG, FLOAT to DOUBLE, or STRING to TIMESTAMP)
2. Different writers -- files produced by different Spark versions, pandas, PyArrow, or other tools may use slightly different Parquet type mappings for the same logical data
3. Nested struct differences -- fields inside struct columns may have different nullability or field ordering across files
4. Decimal precision/scale mismatches -- e.g., DECIMAL(10,2) in some files vs DECIMAL(18,4) in others

APPROACH 1: READ WITH mergeSchema AND REWRITE AS DELTA (RECOMMENDED)

Instead of in-place conversion, read the Parquet files with schema merging enabled and write them out as a new Delta table. The Parquet reader's mergeSchema option is more forgiving -- it can widen types (INT to LONG, FLOAT to DOUBLE) and merge structs with different fields:

df = (spark.read
.option("mergeSchema", "true")
.parquet("abfss://container@storage_account.dfs.core.windows.net/directory_name"))

df.write.format("delta").mode("overwrite").saveAsTable("your_catalog.your_schema.your_table")

If your data is partitioned, add partitionBy:

(df.write.format("delta")
.partitionBy("your_partition_column")
.mode("overwrite")
.saveAsTable("your_catalog.your_schema.your_table"))

Note: this copies the data to a new location, so you will need sufficient storage. The benefit is a clean Delta table with a consistent schema.

APPROACH 2: IDENTIFY THE CONFLICTING FILES

If you want to keep the in-place conversion (no data copy), first identify which files have the conflicting schema so you can fix them:

path = "abfss://container@storage_account.dfs.core.windows.net/directory_name"

files = (spark.read.format("binaryFile")
.load(path + "/*.parquet")
.select("path")
.collect())

schemas = {}
for row in files:
file_path = row["path"]
try:
schema = spark.read.parquet(file_path).schema
schemas[file_path] = schema
except Exception as e:
print(f"Error reading {file_path}: {e}")

# Compare all schemas against the first file
reference_schema = list(schemas.values())[0]
for file_path, schema in schemas.items():
if schema != reference_schema:
print(f"Schema mismatch in: {file_path}")
for ref_field, file_field in zip(reference_schema.fields, schema.fields):
if ref_field != file_field:
print(f" Column difference: {ref_field} vs {file_field}")

Once you identify the problematic files, you can rewrite just those files with the correct schema, then retry CONVERT TO DELTA.

APPROACH 3: SPECIFY AN EXPLICIT SCHEMA AND REWRITE

If you already know the target schema, you can force all files to be read with that schema. Spark will cast compatible types automatically:

from pyspark.sql.types import StructType, StructField, StringType, LongType, DoubleType

target_schema = StructType([
StructField("id", LongType(), True),
StructField("name", StringType(), True),
StructField("amount", DoubleType(), True),
# add all your columns here
])

df = spark.read.schema(target_schema).parquet(
"abfss://container@storage_account.dfs.core.windows.net/directory_name")

df.write.format("delta").mode("overwrite").saveAsTable("your_catalog.your_schema.your_table")

This bypasses the automatic schema merge entirely by telling Spark exactly what types to expect.

APPROACH 4: USE CTAS (CREATE TABLE AS SELECT)

Another clean approach is to use a SQL-based CTAS statement, which lets you read and write in one step:

CREATE OR REPLACE TABLE your_catalog.your_schema.your_table
AS SELECT * FROM parquet.`abfss://container@storage_account.dfs.core.windows.net/directory_name`

DOCUMENTATION REFERENCES

- CONVERT TO DELTA syntax:
https://docs.databricks.com/en/sql/language-manual/delta-convert-to-delta.html

- Convert Parquet to Delta Lake guide:
https://docs.databricks.com/en/delta/convert-to-delta.html

- Delta Lake schema evolution (mergeSchema and autoMerge):
https://docs.databricks.com/en/delta/update-schema.html

SUMMARY

The core issue is that CONVERT TO DELTA does a strict schema merge across all Parquet file footers and cannot automatically widen types the way a DataFrame read can. The spark.databricks.delta.mergeSchema.enabled config only applies to Delta write operations, not to CONVERT TO DELTA.

The most reliable approach is to use spark.read.option("mergeSchema", "true").parquet(...) to read all the files with flexible type widening, then write out a new Delta table. If you share the specific column details from the error message, I can help narrow down exactly which type conflict is causing the failure.

* This reply used an agent system I built to research and draft this response based on the wide set of documentation I have available and previous memory. I personally review the draft for any obvious issues and for monitoring system reliability and update it when I detect any drift, but there is still a small chance that something is inaccurate, especially if you are experimenting with brand new features.

topic CONVERT TO DELTA fails to merge file schema in Data Engineering

CONVERT TO DELTA fails to merge file schema

Re: CONVERT TO DELTA fails to merge file schema

Re: CONVERT TO DELTA fails to merge file schema