To handle the scenario where your pipeline fails after loading some records into the first gold table or if one gold table loads successfully while the second fails, you can implement a failure handling mechanism that ensures already inserted records are not reprocessed when the pipeline is re-run. Here are some steps you can follow:

1. Use Delta Lake for ACID Transactions: Delta Lake provides ACID transactions, which can help ensure that your data is consistent and reliable. If a failure occurs, you can use Delta Lake's transaction log to identify which records have already been processed.

2. Implement Checkpoints: Use checkpoints to save the state of your data processing at various stages. This way, if a failure occurs, you can restart the pipeline from the last successful checkpoint rather than from scratch.

3. Idempotent Writes: Ensure that your write operations are idempotent. This means that re-running the same operation multiple times will not result in duplicate records. You can achieve this by using upsert operations (merge) instead of insert operations.

4. Delta Live Tables (DLT): Consider using Delta Live Tables, which provide built-in capabilities for handling incremental data processing and failure recovery. DLT can automatically manage the state of your data pipeline and ensure that only new or changed data is processed.

5. Repair and Rerun: Utilize the "Repair and Rerun" feature in Databricks jobs. This feature allows you to rerun only the tasks that were impacted by a failure, without reprocessing the entire pipeline. This can save time and resources. You can find more details about this feature in the Databricks blog post titled "Save Time and Money on Data and ML Workflows With 'Repair and Rerun'".