cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Dynamic Continuous Data type and new column arrival adopt in silver in DLT

Denis12345
New Contributor

My source is : Oracle db - Golden gate - kafka - Flink - S3 - bronze - silver - gold 

It's a live cdc data, in S3 we have delta tables in string format, bronze also having string format for all tables, it's a DLT pipeline 

Now we have to map datatypes and table structure like existing columns and new column arrival in silver dynamically continuously without delay of Microsoft seconds, pipelines and jobs we can not stop.

Note - we have one metadata snapshot table in S3, which we can consume to map in silver

Kindly advise how to achieve this

0 REPLIES 0