Hey everyone ๐
I totally get the frustration of dealing with these complex data layers, but don't worry, you're in the right place for some guidance! Accessing that final delta table through a web interface can indeed be a bit tricky, but it's not an unsolvable puzzle.
Your plan to use a web script running Spark SQL sounds pretty solid. This way, you can leverage the power of Spark to handle the heavy lifting and get the aggregated data. Storing the results in a database like MongoDB and then displaying it in the web UI is a sensible approach too, as it can help with faster retrieval and a smoother user experience.
However, if you're looking for a best practice solution, you might want to consider a microservices architecture. This could involve building a dedicated service that handles the interaction between your Spark cluster and the web UI. By decoupling these components, you could achieve better scalability and maintainability.
I am really glad that you are asking such relevant and important questions on this forum. Mobile development is still gaining popularity right now, and I would recommend you check out this resource to better understand your problem: Mobile App Development Industry: Explore Top Challenges in 2023. Let me know if this has helped you in any way.
User16826994223, about your question, kudos for your initiative! If you're going the Spark SQL route, remember to optimize your queries for performance, and perhaps consider caching if applicable. Also, explore visualization libraries to make your web interface more user-friendly.
