cancel
Showing results for 
Search instead for 
Did you mean: 
Get Started Discussions
Start your journey with Databricks by joining discussions on getting started guides, tutorials, and introductory topics. Connect with beginners and experts alike to kickstart your Databricks experience.
cancel
Showing results for 
Search instead for 
Did you mean: 

CDC for Unstructured data

Phani1
Valued Contributor II

Hi All,

how we can handle CDC for unstructured data in Databricks. What are some best practices we should follow to make this work effectively?

Regards,

Phani

1 REPLY 1

filipniziol
Contributor

Hi @Phani1 ,

Handling CDC for unstructured data—such as audio, images, or video files—in Databricks involves efficiently detecting and processing changes to these files as they occur.

Here's how you can approach this:

  • Use Databricks Autoloader: Autoloader can incrementally and efficiently process new and changed files from cloud storage like Azure Data Lake. It uses a checkpointing mechanism to track which files have been processed.
  • Since delta format allows binary data can be stored in a Delta table, you may read the files with its metadata attribures (modification timestamps, checksums) and store as Delta table. The files uploaded by the autoloader may be then compared against their checksums to recognize duplicates etc.

 

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group