https://community.databricks.com/t5/get-started-discussions/cdc-for-unstructured-data/m-p/95973#M4489 <P>Hi&nbsp;<a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/36892">@Phani1</a>&nbsp;,</P><P>Handling CDC for unstructured data—such as audio, images, or video files—in Databricks involves efficiently detecting and processing changes to these files as they occur.</P><P>Here's how you can approach this:</P><UL><LI>Use Databricks Autoloader: Autoloader can incrementally and efficiently process new and changed files from cloud storage like Azure Data Lake. It uses a checkpointing mechanism to track which files have been processed.</LI><LI>Since delta format allows&nbsp;binary data can be stored in a Delta table, you may read the files with its metadata attribures (modification timestamps, checksums) and store as Delta table. The files uploaded by the autoloader may be then compared against their checksums to recognize duplicates etc.</LI></UL><P>&nbsp;</P> Thu, 24 Oct 2024 13:11:23 GMT filipniziol 2024-10-24T13:11:23Z https://community.databricks.com/t5/get-started-discussions/cdc-for-unstructured-data/m-p/95969#M4488 <P>Hi All,</P><P><SPAN>how we can handle CDC for unstructured data in Databricks. What are some best practices we should follow to make this work effectively?</SPAN></P><P>Regards,</P><P>Phani</P> Thu, 24 Oct 2024 12:48:44 GMT https://community.databricks.com/t5/get-started-discussions/cdc-for-unstructured-data/m-p/95969#M4488 Phani1 2024-10-24T12:48:44Z https://community.databricks.com/t5/get-started-discussions/cdc-for-unstructured-data/m-p/95973#M4489 <P>Hi&nbsp;<a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/36892">@Phani1</a>&nbsp;,</P><P>Handling CDC for unstructured data—such as audio, images, or video files—in Databricks involves efficiently detecting and processing changes to these files as they occur.</P><P>Here's how you can approach this:</P><UL><LI>Use Databricks Autoloader: Autoloader can incrementally and efficiently process new and changed files from cloud storage like Azure Data Lake. It uses a checkpointing mechanism to track which files have been processed.</LI><LI>Since delta format allows&nbsp;binary data can be stored in a Delta table, you may read the files with its metadata attribures (modification timestamps, checksums) and store as Delta table. The files uploaded by the autoloader may be then compared against their checksums to recognize duplicates etc.</LI></UL><P>&nbsp;</P> Thu, 24 Oct 2024 13:11:23 GMT https://community.databricks.com/t5/get-started-discussions/cdc-for-unstructured-data/m-p/95973#M4489 filipniziol 2024-10-24T13:11:23Z