4 weeks ago - last edited 4 weeks ago
Hi Team,
I would like to know difference between _RAW tables and _APPEND_RAW tables of Bronze-Layer.
As both are STREAMING Tables then why we need 2 separate tables.
Note: we are following Medalion Architecture. Also above tables are created via Delta Live Tables pipeline. so they are basically DLT Tables.
Thanks
Devsql
3 weeks ago - last edited 3 weeks ago
Hi @Devsql,
_RAW Tables:
_RAW
tables represent the raw, unprocessed data ingested into your system. They typically contain the original data as it arrives, without any transformations or modifications._RAW
tables are useful for auditing, lineage tracking, and maintaining a historical record of the raw data._APPEND_RAW Tables:
_APPEND_RAW
tables also store raw data, but they allow for both inserts and updates. In addition to new records, they can capture changes to existing records (updates)._RAW
tables, _APPEND_RAW
tables can handle both insert and update operations._APPEND_RAW
tables are a better fit.Why Separate Tables?
_RAW
and _APPEND_RAW
tables depends on your use case and architecture._RAW
tables._APPEND_RAW
tables.Both types of tables are created via the Delta Live Tables pipeline, making them part of the Delta Lake ecosystem.
Feel free to ask if you need further clarification or have additional questions! ๐
4 weeks ago
I don't exactly understand your question, so let me try to give you a generic answer. You don't need to do anything, if you're fine with working with one table, then just go with one.
An append-only table, as the name suggests, will only contain insert operations. By using the table property "delta.appendOnly" you can also force it.
And "a not-only append" table, obviously, might contain next to inserts, also updates and deletes.
In general, there's no right or wrong answer, as it highly depends on your use case/architecture.
4 weeks ago
Hi @Witold , I updated last line of my above post, added below lines: Also above tables are created via Delta Live Tables pipeline. so they are basically DLT Tables. Hope this gives you proper idea.
4 weeks ago
Hi @Kaniz_Fatma , I saw your replies to other posts, so thought to ask you....would you like to help me on this...!!!
3 weeks ago - last edited 3 weeks ago
Hi @Devsql,
_RAW Tables:
_RAW
tables represent the raw, unprocessed data ingested into your system. They typically contain the original data as it arrives, without any transformations or modifications._RAW
tables are useful for auditing, lineage tracking, and maintaining a historical record of the raw data._APPEND_RAW Tables:
_APPEND_RAW
tables also store raw data, but they allow for both inserts and updates. In addition to new records, they can capture changes to existing records (updates)._RAW
tables, _APPEND_RAW
tables can handle both insert and update operations._APPEND_RAW
tables are a better fit.Why Separate Tables?
_RAW
and _APPEND_RAW
tables depends on your use case and architecture._RAW
tables._APPEND_RAW
tables.Both types of tables are created via the Delta Live Tables pipeline, making them part of the Delta Lake ecosystem.
Feel free to ask if you need further clarification or have additional questions! ๐
3 weeks ago
Thank you very much @Kaniz_Fatma for this excellent answer.
Excited to expand your horizons with us? Click here to Register and begin your journey to success!
Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!