Getting concurrent Append exception after upgrading to 14.3 runtime LTS

[Dilisha]

Hi. I have upgraded to 14.3 recently and facing the same issue in production.

The below is my update statement

UPDATE table1 
SET starttime = getArgument("time1"), endtime = null  
WHERE ConnectionID=getArgument("ConnectionInfo") AND GroupName = getArgument("JobGroup")
    AND EXISTS (SELECT date FROM table1 WHERE ConnectionID=getArgument("ConnectionInfo") AND GroupName = getArgument("JobGroup") AND endtime IS NOT NULL AND getArgument("RunType") = 'start')
    AND IsSyncRequired = true

below is the error we are facing :

"runError": "ConcurrentAppendException: Files were added to partition [ConnectionID=con1, GroupName=group2] by a concurrent update. Please try the operation again.

{
"timestamp": 1719876118108,
"userId": "7939453779672853",
"userName": "x",
"operation": "UPDATE",
"operationParameters": {
"predicate": [
"(((ConnectionID#99116459 = getArgument(ConnectionInfo)) AND (GroupName#99116455 = getArgument(JobGroup))) AND (isnotnull(scalar-subquery#99116271 []) AND IsSyncRequired#99116460))"
]
},
"job": {
"jobId": "x",
"jobName": "x",
"runId": "x",
"jobOwnerId": "x",
"triggerType": "manual"
},
"notebook": {
"notebookId": "x"
},
"clusterId": "x",
"readVersion": 1513710,
"isolationLevel": "WriteSerializable",
"isBlindAppend": false,
"operationMetrics": {
"numRemovedFiles": "1",
"numRemovedBytes": "1732",
"numCopiedRows": "0",
"numDeletionVectorsAdded": "0",
"numDeletionVectorsRemoved": "0",
"numAddedChangeFiles": "0",
"executionTimeMs": "9837",
"numDeletionVectorsUpdated": "0",
"scanTimeMs": "7487",
"numAddedFiles": "1",
"numUpdatedRows": "1",
"numAddedBytes": "1631",
"rewriteTimeMs": "2309"
},
"tags": {
"noRowsCopied": "false",
"delta.rowTracking.preserved": "false",
"restoresDeletedRows": "false"
},
"engineInfo": "Databricks-Runtime/14.3.x-scala2.12",
"txnId": "xx"
}

Please note that my group1 update is failing with error on group2 conflicting commit when ideally both of them should go to different partitions.

The partition columns are both ConnectionID and GroupName. 

Can someone please help me understand when my where clause specifies both, why is it still giving me ConcurrentAppendException?

[Kaniz_Fatma]

Hi, @Dilisha, When creating a table, you need to specify the fields (columns) explicitly. In your case, you’re creating a table named outdoorProductsRaw. Make sure to define the columns within the CREATE TABLE statement. For example:

1. CREATE TABLE IF NOT EXISTS outdoorProductsRaw (
       column_a STRING,
       number INT
   ) USING csv OPTIONS (
       path "/mnt/training/online_retail/data-001/data.csv",
       header "true"
   );
   

   Replace column_a and number with the actual column names from your dataset.

2. If you’re using partitioned tables, ensure that you include a partition column. For instance:

   CREATE TABLE IF NOT EXISTS outdoorProductsRaw (
       column_a STRING,
       number INT
   ) USING csv
   PARTITIONED BY (column_a)
   OPTIONS (
       path "/mnt/training/online_retail/data-001/data.csv",
       header "true"
   );
   

3. Verify that you have the necessary permissions for creating tables and writing to the specified location. Permissions on the Unity Catalog (if applicable) should also be considered.

4. If you encounter issues related to a managed table already existing, you can use the following workaround:

   dbutils.fs.rm("dbfs:/user/hive/warehouse/SomeData/", True)
   

   This command removes the existing table before re-creating it.

If you encounter any further issues, feel free to ask for additional assistance! 😊