Thank you so much for the detailed and thoughtful response!
I’m very new to the real-time data processing world, so it will take me some time to fully understand and explore the advanced concepts you mentioned. That said, your explanation and suggestions are extremely helpful, and I really appreciate the effort you put into sharing your knowledge. I may follow up with a few questions once I’ve had more time to digest everything and implement that in my code.