https://community.databricks.com/t5/data-engineering/best-practices-near-real-time-processing/m-p/65755#M32916 <P>HI All,</P><P>We are ingesting 1000 files in json format and different sizes per minute. DLT is in continuous mode. Unity Catalog is enabled workspace.&nbsp;&nbsp;We are using the default setting of Autoloader (Directory Listing) and Silver has CDC as well.</P><P>We aim to have a near real-time processing. What are the best practices to do this?&nbsp;</P><P>Cheers,</P> Mon, 08 Apr 2024 06:57:53 GMT Gilg 2024-04-08T06:57:53Z https://community.databricks.com/t5/data-engineering/best-practices-near-real-time-processing/m-p/65755#M32916 <P>HI All,</P><P>We are ingesting 1000 files in json format and different sizes per minute. DLT is in continuous mode. Unity Catalog is enabled workspace.&nbsp;&nbsp;We are using the default setting of Autoloader (Directory Listing) and Silver has CDC as well.</P><P>We aim to have a near real-time processing. What are the best practices to do this?&nbsp;</P><P>Cheers,</P> Mon, 08 Apr 2024 06:57:53 GMT https://community.databricks.com/t5/data-engineering/best-practices-near-real-time-processing/m-p/65755#M32916 Gilg 2024-04-08T06:57:53Z