https://community.databricks.com/t5/data-engineering/help-design-my-streaming-pipeline/m-p/55865#M30448 <P>###Data Source<BR />- AWS RDS<BR />- Database migration tasks have been created using AWS DMS<BR />- Relevant cdc information is being stored in a specific bucket in S3</P><P>### Data frequency<BR />- Once a day (but not sure when, sometime after 6pm)</P><P>### Development environment<BR />- databricks<BR />- Delat Live Table from databricks</P><P>### Data Status<BR />- CLOSE_DT, CURR_F_CD, CURR_T_CD are PK, JOIN conditions<BR />- CLOSE_DT is DATE type<BR />- Data comes in from source(=RDS) once a day on weekdays.<BR />- This data is written as a cdc to S3 via AWS DMS</P><P>### Processing requirements<BR />- No data comes into source on non-weekday holidays, but must be matched to the most recent data.<BR />- Data comes in once a day on weekdays, and the presence or absence of a specific CLOSE_DT can be used to determine if data came in today or not.<BR />- For example, let's say today is 2023-12-28.<BR />- You don't know when data with a CLOSE_DT of 2023-12-28 will come in today.<BR />- So until the data comes in, you create the 2023-12-28 data from the most recent 2023-12-27 data.<BR />- When the 2023-12-28 data comes in, the data is swapped.<BR />- No data comes in at all on holidays, so data must be generated with the most recent data each day</P> Thu, 28 Dec 2023 00:54:24 GMT rt-slowth 2023-12-28T00:54:24Z https://community.databricks.com/t5/data-engineering/help-design-my-streaming-pipeline/m-p/55865#M30448 <P>###Data Source<BR />- AWS RDS<BR />- Database migration tasks have been created using AWS DMS<BR />- Relevant cdc information is being stored in a specific bucket in S3</P><P>### Data frequency<BR />- Once a day (but not sure when, sometime after 6pm)</P><P>### Development environment<BR />- databricks<BR />- Delat Live Table from databricks</P><P>### Data Status<BR />- CLOSE_DT, CURR_F_CD, CURR_T_CD are PK, JOIN conditions<BR />- CLOSE_DT is DATE type<BR />- Data comes in from source(=RDS) once a day on weekdays.<BR />- This data is written as a cdc to S3 via AWS DMS</P><P>### Processing requirements<BR />- No data comes into source on non-weekday holidays, but must be matched to the most recent data.<BR />- Data comes in once a day on weekdays, and the presence or absence of a specific CLOSE_DT can be used to determine if data came in today or not.<BR />- For example, let's say today is 2023-12-28.<BR />- You don't know when data with a CLOSE_DT of 2023-12-28 will come in today.<BR />- So until the data comes in, you create the 2023-12-28 data from the most recent 2023-12-27 data.<BR />- When the 2023-12-28 data comes in, the data is swapped.<BR />- No data comes in at all on holidays, so data must be generated with the most recent data each day</P> Thu, 28 Dec 2023 00:54:24 GMT https://community.databricks.com/t5/data-engineering/help-design-my-streaming-pipeline/m-p/55865#M30448 rt-slowth 2023-12-28T00:54:24Z