Hi @Phani1 ,
Handling CDC for unstructured dataโsuch as audio, images, or video filesโin Databricks involves efficiently detecting and processing changes to these files as they occur.
Here's how you can approach this:
- Use Databricks Autoloader: Autoloader can incrementally and efficiently process new and changed files from cloud storage like Azure Data Lake. It uses a checkpointing mechanism to track which files have been processed.
- Since delta format allows binary data can be stored in a Delta table, you may read the files with its metadata attribures (modification timestamps, checksums) and store as Delta table. The files uploaded by the autoloader may be then compared against their checksums to recognize duplicates etc.
