Databricks Community

VaDim

As per Databricks documentation when I define the class that extends `StatefulProcessor` in a Notebook everything works ok however, execution fails with ModuleNotFound error as soon as the class definition is moved to a file (module) of it's own in a...

VaDim · 09-08-2022

I can't seem to make it work as I keep getting:DeltaInvariantViolationException: NOT NULL constraint violated for column: dl_id.

VaDim · 09-08-2022

I haven't tried but I suspect it will fail with the same message on INSERT because uc.dl_id is NULL for some rows and `whenNotMatchedInsertAll` will attempt to insert a value for dl_id field instead of generating one (as if it has been user provided...

VaDim · 09-08-2022

`ec` - always. `uc` - not. because `uc` might contain new data that needs appended.

VaDim · 09-08-2022

I use 11.2 runtime. identity column created as:dl_id BIGINT NOT NULL GENERATED ALWAYS AS IDENTITYand the pyspark code is an upsert:t.alias("ec").merge(df.alias("uc"), "ec.dl_id = uc.dl_id") \ .whenMatchedUpdateAll() \ .whenN...

Databricks Community

User Stats

User Activity

ModuleNotFound error when using transformWithStateInPandas via a class defined outside the notebook

Are MERGE INTO inserts supported when the delta table has an identity column ?

Re: Are MERGE INTO inserts supported when the delta table has an identity column ?

Re: Are MERGE INTO inserts supported when the delta table has an identity column ?

Re: Are MERGE INTO inserts supported when the delta table has an identity column ?