DeltaColumnMappingUnsupportedException' when performing 'Full refresh all' on DLT pipeline

[429957]

Trigger:

Perform 'Full refresh all' on a DLT pipeline (new or existing). The existing DLT table already existed beforehand.

Issue:

Getting the error 'DeltaColumnMappingUnsupportedException' during "Setting up tables" stage.

```com.databricks.sql.transaction.tahoe.DeltaColumnMappingUnsupportedException:

Schema change is detected:

old schema:

root

new schema:

root

|-- Field 1: string (nullable = true)

|-- Field 2: string (nullable = true)

|-- Field 3: timestamp (nullable = true)

Schema changes are not allowed during the change of column mapping mode.

```

Setup:

The DLT pipeline is configured to run a python notebook that contains something like this:

```

@dlt.table(

  comment="Raw table",

  table_properties={

    "delta.minReaderVersion": "2",

    "delta.minWriterVersion": "5",

    "delta.columnMapping.mode": "name",

  },

)

def raw_table():

  return (spark.read.format(file_type) 

    .option("inferSchema", false) 

    .option("header", true) 

    .load(some_file_location))

```

Appreciate any insights to this issue. Thanks.

[Anonymous]

@Raeger Tay​ :

The error message indicates that a schema change has been detected while changing the column mapping mode. It seems like you are trying to change the column mapping mode from the default (position) to the "name" mode, which maps columns based on their names instead of their positions.

When changing the column mapping mode, you need to ensure that the schema of the table remains the same. In your case, the old schema had only one column (with no name) while the new schema has three columns with names. This is why you are getting the schema change error.

To fix this issue, you can try one of the following options:

1. Use the default column mapping mode (position) instead of the "name" mode. This way, you don't need to worry about schema changes while changing the column mapping mode.
2. Create a new DLT pipeline with the desired schema and column mapping mode. This will avoid any conflicts with the existing table schema.
3. If you need to modify the existing table schema, you can do so by running ALTER TABLE commands in a SQL notebook. However, you need to ensure that you don't change the schema while changing the column mapping mode.

Once you have fixed the schema issue, you can try running the "Full refresh all" again.