https://community.databricks.com/t5/data-engineering/near-real-time-processing-with-cdc-from-snowflake-to-databricks/m-p/106508#M42502 <P>it is like latency vs complexity and cost. you have to choose for yourself <span class="lia-unicode-emoji" title=":slightly_smiling_face:">🙂</span> for me option A sounds reasonable</P> Tue, 21 Jan 2025 14:36:40 GMT saurabh18cs 2025-01-21T14:36:40Z https://community.databricks.com/t5/data-engineering/near-real-time-processing-with-cdc-from-snowflake-to-databricks/m-p/106507#M42501 <DIV class=""><SPAN>Hi</SPAN><SPAN>&nbsp;</SPAN><SPAN>I would like to configure near real time streaming on Databricks to process data as soon as a new data finish processing on snowflake e.g. with DLT pipelins and Auto Loader. Which option would be better for this setup?</SPAN></DIV><DIV class=""><DIV class=""><DIV class=""><DIV class="">&nbsp;</DIV><DIV class="">Option A)</DIV><DIV class=""><SPAN>Export the Snowpark DataFrame to Databricks to an external cloud storage (e.g. S3 as parquet).</SPAN></DIV><DIV class="">&nbsp;</DIV><DIV class="">Option B)</DIV><DIV class=""><SPAN>&nbsp;use&nbsp;apache iceberg&nbsp;</SPAN><SPAN>with polaris and configure from Databricks in order to read that information.</SPAN></DIV></DIV></DIV></DIV> Tue, 21 Jan 2025 14:45:31 GMT https://community.databricks.com/t5/data-engineering/near-real-time-processing-with-cdc-from-snowflake-to-databricks/m-p/106507#M42501 abelian-grape 2025-01-21T14:45:31Z https://community.databricks.com/t5/data-engineering/near-real-time-processing-with-cdc-from-snowflake-to-databricks/m-p/106508#M42502 <P>it is like latency vs complexity and cost. you have to choose for yourself <span class="lia-unicode-emoji" title=":slightly_smiling_face:">🙂</span> for me option A sounds reasonable</P> Tue, 21 Jan 2025 14:36:40 GMT https://community.databricks.com/t5/data-engineering/near-real-time-processing-with-cdc-from-snowflake-to-databricks/m-p/106508#M42502 saurabh18cs 2025-01-21T14:36:40Z