Not able to insert into identity column through spark

Shivaprasad
Contributor

I have a delta table with identity column and not able to insert data using spark. I am using 15.4LTS. any idea what needed to be done

 

Table Name : account

Column Def: account_dimension_id BIGINT GENERATED BY DEFAULT AS IDENTITY,

df = spark.read.format("csv").load("abfss://databricks-storage@sa14127e1dv0101.dfs.core.windows.net/catalogs/data-relationship-management-drm/onprem_acct_dimension.csv");

df.write.format("delta").mode("append").insertInto("account")

I am able to insert through sql but not through spark

file has 7 columns but table has 8 columns including the identity column

 Error: error: [DELTA_INSERT_COLUMN_ARITY_MISMATCH] Cannot write to 'data_relationship_managment_pdz.data_relationship_managment_schema.account', not enough data columns; target table has 8 column(s) but the inserted data has 7 column(s) SQLSTATE: 42802