The error was more related to a lot of stuff on the AWS side, so we deleted and cleared the SQS and SNS. we also used the CloudFilesAWSResourceManager

val manager = CloudFilesAWSResourceManager
    .newManager
    .option("path", filePath)
    .create()
   manager.setUpNotificationServices(notificationServices)