ACID properties in delta?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-05-2023 04:12 PM
How are locks maintained within a Delta Lake? For instance, lets say there are 2 simple tables, customer_details and say orders. Lets say I am running a job that will say insert an order in the orders table for say $100 for a specific customerId, it should go and update (increment) the customer_details table with the order_count value by 1 and also update the order_value details by 100. Note that until the the orders table is fully updated with all the information, the customer_details table should not be updated and also, once the orders table is inserted/deleted, the customer_details table HAS to be updated with the right counts and dollars. In a traditional DB, we have this concept of savepoints where we can combine multiple CRUD operations as a 'transaction' and either fail (rollback?) everything or commit everything to the DB. How is this possible in a delta environment? While ACID capabilities exist at an individual table level, how can this be achieved in a delta lake ? (Kindly note that updating the customer_details table after the fact as a batch job is a solution but this is just a simple use case I have posted. There is a good chance that an "order" can also require data to be stored in multiple tables). Thanks in advance..