https://community.databricks.com/t5/get-started-discussions/best-practices-for-implementing-early-arriving-fact-handling/m-p/84305#M3683 <P>&nbsp;</P><P>Hi All,</P><P>Can you please share us the best practices for implementing early arriving fact handling in databricks for streaming data processed in near real time using structured streaming.</P><P>There are many ways to handle this use case in batch/mini batch. Specially we are looking for best practices to handle this use case using structured streaming in near real time.</P><P>example:</P><P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Phani1_0-1724754033290.png" style="width: 400px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/10606iB097B31B5372ABB0/image-size/medium/is-moderation-mode/true?v=v2&amp;px=400" role="button" title="Phani1_0-1724754033290.png" alt="Phani1_0-1724754033290.png" /></span></P><P>&nbsp;</P><P>&nbsp;</P><P><STRONG><U>Example of early arriving fact:</U></STRONG></P><P>Please refer to the below tables explaining early arriving fact scenarios.</P><UL><LI>One record is received (highlighted in red) in SalesDetail transaction data where corresponding customer (C4) is not loaded into DimCustomer dimension yet.</LI><LI>The data for fact (FactSalesDetail) table arrived earlier than corresponding dimension (C4 in DimCustomer) data.</LI></UL><P>Regards,</P><P>Phani</P> Tue, 27 Aug 2024 10:21:08 GMT Phani1 2024-08-27T10:21:08Z https://community.databricks.com/t5/get-started-discussions/best-practices-for-implementing-early-arriving-fact-handling/m-p/84305#M3683 <P>&nbsp;</P><P>Hi All,</P><P>Can you please share us the best practices for implementing early arriving fact handling in databricks for streaming data processed in near real time using structured streaming.</P><P>There are many ways to handle this use case in batch/mini batch. Specially we are looking for best practices to handle this use case using structured streaming in near real time.</P><P>example:</P><P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Phani1_0-1724754033290.png" style="width: 400px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/10606iB097B31B5372ABB0/image-size/medium/is-moderation-mode/true?v=v2&amp;px=400" role="button" title="Phani1_0-1724754033290.png" alt="Phani1_0-1724754033290.png" /></span></P><P>&nbsp;</P><P>&nbsp;</P><P><STRONG><U>Example of early arriving fact:</U></STRONG></P><P>Please refer to the below tables explaining early arriving fact scenarios.</P><UL><LI>One record is received (highlighted in red) in SalesDetail transaction data where corresponding customer (C4) is not loaded into DimCustomer dimension yet.</LI><LI>The data for fact (FactSalesDetail) table arrived earlier than corresponding dimension (C4 in DimCustomer) data.</LI></UL><P>Regards,</P><P>Phani</P> Tue, 27 Aug 2024 10:21:08 GMT https://community.databricks.com/t5/get-started-discussions/best-practices-for-implementing-early-arriving-fact-handling/m-p/84305#M3683 Phani1 2024-08-27T10:21:08Z https://community.databricks.com/t5/get-started-discussions/best-practices-for-implementing-early-arriving-fact-handling/m-p/85833#M4043 <P><SPAN>Greetings Team, I would like to inquire if any of you have suggestions regarding the query.</SPAN></P> Wed, 28 Aug 2024 14:41:54 GMT https://community.databricks.com/t5/get-started-discussions/best-practices-for-implementing-early-arriving-fact-handling/m-p/85833#M4043 Phani1 2024-08-28T14:41:54Z