Databricks Community

Anatoly · 10-30-2022

Hi!I have a delta table and a process that reading a stream from this table.I need to drop the NOT NULL constraint from some of the columns of this table.The first drop command does not affect the reading stream.But the second command results in erro...

Anatoly · 10-31-2022

@Kaniz Fatma The command used to drop constraint is:ALTER TABLE temp_schema.temp_stream_table ALTER COLUMN a DROP NOT NULL;This means, I want to make the column nullable, not to drop it.The error appears on readStream step, not on drop constraint.

Anatoly · 10-31-2022

DBR 10.4 LTS.Also tested on 11.3 LTS - same result.

Anatoly · 10-31-2022

Yes, for field "b". nullable=true vs nullable=false

Anatoly · 10-31-2022

Hi @Kaniz Fatma ,I cleanup the checkpoint directory in rows 71-12, it should be empty.Also no new data is inserted into table, the error is on read.

Databricks Community

User Stats

User Activity

"Detected schema change" error while reading from delta table in streaming after applying "ALTER COLUMN DROP NOT NULL" to more than one columns.

Re: "Detected schema change" error while reading from delta table in streaming after applying "ALTER COLUMN DROP NOT NULL" to more than one columns.

Re: "Detected schema change" error while reading from delta table in streaming after applying "ALTER COLUMN DROP NOT NULL" to more than one columns.

Re: "Detected schema change" error while reading from delta table in streaming after applying "ALTER COLUMN DROP NOT NULL" to more than one columns.

Re: "Detected schema change" error while reading from delta table in streaming after applying "ALTER COLUMN DROP NOT NULL" to more than one columns.