topic Re: How to access final delta table for web application or interface. in Data Engineering

How to access final delta table for web application or interface.

User16826994223 — Fri, 25 Jun 2021 17:25:47 GMT

I have a final layer of the gold delta table, that has final aggregated data from silver data . I want to access this final layer of data through the WEB interface

I think I need to write a web script that would run the spark SQL behind to get the data. and then i Can write the result set data in soem table like mango db and then show in web ui

Is there a known best practice solution?

Re: How to access final delta table for web application or interface.

User16826994223 — Fri, 25 Jun 2021 17:27:08 GMT

The real answer depends on your requirements regarding latency, the number of data located - HDFS/S3/...), etc. Possible approaches are:

Read data directly using the Delta Standalone Reader library for JVM, or via delta-rs library that works with Rust/Python/Ruby

Re: How to access final delta table for web application or interface.

Altay — Fri, 17 Mar 2023 11:07:11 GMT

Is there any update on this topic?

Thank you

Re: How to access final delta table for web application or interface.

stefanhieslas11 — Tue, 15 Aug 2023 12:08:28 GMT

Hey everyone 🙂

I totally get the frustration of dealing with these complex data layers, but don't worry, you're in the right place for some guidance! Accessing that final delta table through a web interface can indeed be a bit tricky, but it's not an unsolvable puzzle.

Your plan to use a web script running Spark SQL sounds pretty solid. This way, you can leverage the power of Spark to handle the heavy lifting and get the aggregated data. Storing the results in a database like MongoDB and then displaying it in the web UI is a sensible approach too, as it can help with faster retrieval and a smoother user experience.

However, if you're looking for a best practice solution, you might want to consider a microservices architecture. This could involve building a dedicated service that handles the interaction between your Spark cluster and the web UI. By decoupling these components, you could achieve better scalability and maintainability.

I am really glad that you are asking such relevant and important questions on this forum. Mobile development is still gaining popularity right now, and I would recommend you check out this resource to better understand your problem: Mobile App Development Industry: Explore Top Challenges in 2023. Let me know if this has helped you in any way.

User16826994223, about your question, kudos for your initiative! If you're going the Spark SQL route, remember to optimize your queries for performance, and perhaps consider caching if applicable. Also, explore visualization libraries to make your web interface more user-friendly.

Re: How to access final delta table for web application or interface.

shadowinc — Sat, 02 Nov 2024 09:36:46 GMT

@stefanhieslas11 Thanks for your input. However, are there other approaches to avoid staging databases like Cosmos or Mongo DB as input to web applications and directly fetch from delta tables while maintaining the read performance? Much appreciated.

Re: How to access final delta table for web application or interface.

h_h_ak — Sat, 02 Nov 2024 22:43:27 GMT

You can also use direct statement execution from databricks: https://docs.databricks.com/api/workspace/statementexecution