How to Overwrite Using pyspark's JDBC without loosing constraints on table columns

Abeeya — Fri, 01 Apr 2022 11:57:00 GMT

Hello,

My table has primary key constraint on a perticular column, Im loosing primary key constaint on that column each time I overwrite the table , What Can I do to preserve it? Any Heads up would be appreciated

Tried Below

df.write.option("truncate", "true").jdbc(url=DATABASE_URL, table=DATABASE_TABLE, mode="overwrite", properties=DATABASE_PROPERTIES)

But It errored out as below when a new column was added to dataframe which was not there in table

AnalysisException: Column "new_col" not found in schema Some

Re: How to Overwrite Using pyspark's JDBC without loosing constraints on table columns

Hubert-Dudek — Fri, 01 Apr 2022 12:06:23 GMT

@Abeeya . , Mode "truncate", is correct to preserve the table. However, when you want to add a new column (mismatched schema), it wants to drop it anyway.

topic Re: How to Overwrite Using pyspark's JDBC without loosing constraints on table columns in Data Engineering

How to Overwrite Using pyspark's JDBC without loosing constraints on table columns

Re: How to Overwrite Using pyspark's JDBC without loosing constraints on table columns