I have a similar situation in my data quality check process. During this stage, I frequently find errors or potential issues that can stop the pipeline. Each of these errors requires manual intervention, which might involve making edits or supplying justifications for the discrepancies. Once all the issues are resolved, the pipeline can resume its operation without any problems.

I was considering two options:

1. Using Databricks.
2. Pushing the data to AWS DynamoDB and get the response back to continue the process.

What are your thoughts on these options?

Context: it is a multi-tenancy process with many clients