pyspark delta table schema evolution

zyang — Fri, 22 Jul 2022 06:42:33 GMT

I am using the schema evolution in the delta table and the code is written in databricks notebook.

 df.write
        .format("delta")
        .mode("append")
        .option("mergeSchema", "true")
        .partitionBy("date")
        .save(path)

But I still got the error below. Is it correct to define the schema and enable the mergeSchema at the same time?

AnalysisException: The specified schema does not match the existing schema at path.
== Specified ==
 
    root
    -- A: string (nullable = false)
    -- B: string (nullable = true)
    -- C: long (nullable = true)
    
== Existing ==
    root
    -- A: string (nullable = true)
    -- B: string (nullable = true)
    -- C: long (nullable = true)
 
== Differences==
- Field A is non-nullable in specified schema but nullable in existing schema.
 
If your intention is to keep the existing schema, you can omit the
schema from the create table command. Otherwise please ensure that
the schema matches.

Re: pyspark delta table schema evolution

Noopur_Nigam — Tue, 30 Aug 2022 21:04:11 GMT

Hi @z yang Please provide the df creation code as well to understand the complete exception and scenario.

topic pyspark delta table schema evolution in Data Engineering

pyspark delta table schema evolution

Re: pyspark delta table schema evolution