- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-13-2024 09:48 AM
I have a delta table with identity column and not able to insert data using spark. I am using 15.4LTS. any idea what needed to be done
Table Name : account
Column Def: account_dimension_id BIGINT GENERATED BY DEFAULT AS IDENTITY,
df = spark.read.format("csv").load("abfss://databricks-storage@sa14127e1dv0101.dfs.core.windows.net/catalogs/data-relationship-management-drm/onprem_acct_dimension.csv");
df.write.format("delta").mode("append").insertInto("account")
I am able to insert through sql but not through spark
file has 7 columns but table has 8 columns including the identity column
Error: error: [DELTA_INSERT_COLUMN_ARITY_MISMATCH] Cannot write to 'data_relationship_managment_pdz.data_relationship_managment_schema.account', not enough data columns; target table has 8 column(s) but the inserted data has 7 column(s) SQLSTATE: 42802