- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-19-2025 09:13 AM
Hello Rjdudley,
Thanks for your answer! This is really appreciate.
I think I understand your point, and maybe I've described the situation not correctly.
I've said I would have a bunch of 300 tables, yes, but it is true that there is a "structure" behind:
. Let's say - currently in Oracle - we have 3 layers (simplifying): Data Warehouse tables, DataMart tables and reporting tables.
Each layer is based on the data of the previous layer.
Currently in Oracle (trying to describe in a simple way), the PL/SQL is just there to orchestrate the loading of the tables. We are passing to it only statements like "INSERT INTO mytable SELECT * FROM myview", statements which are prepare already in a kind of dictionary table.
The PL/SQL just need to parse this dictionary table, check whether a table should be refreshed or not, and if yes execute the statement.
This is this kind of logic I was thinking to reproduce within Databricks, via PySpark.
In this case, is this logic "acceptable" ?
My understanding is that the layer we call currently "Data Warehouse" is the bronze layer in Databricks, the "DataMart" is the silver layer and the "reporting tables" are the gold layer.
Is it correct ?
Thanks!
Note: Yes, I've registered to some Databricks Academy courses, I try to make it work with my calendar...