Is there a way to get cdc data from salesforce to databricks? Can a smart pipeline be built to get near real time data from salesforce into delta lake?

[Ruby8376]

Currently, we have daily batch running to extract data from salesforce into csv file (adls) which is further copied to delta tables for transformation. We are now looking to implement a solution which can extract real-time data changes on salesforce and load it on databricks for further transformations.

can somebody help with this??

[Ruby8376]

@Werner Stinckens​ can you help?

[Ruby8376]

@Prabakar Ammeappin​ can you help on this?

[Prabakar]

Hi @Ruby Rubi I don't have an information on this. However let's see if @Werner Stinckens or @Hubert Dudek can answer you.

[-werners- · Accepted solution]

I never worked on Salesforce myself, but I do know there are several export possibilities.

https://developer.salesforce.com/docs/atlas.en-us.change_data_capture.meta/change_data_capture/cdc_i...

https://developer.salesforce.com/docs/atlas.en-us.240.0.api_streaming.meta/api_streaming/intro_strea...

Hard to tell what exactly is an option for your environment.

[daniel_sahal]

On Azure you can try using SAP CDC connector for Data Factory:

https://learn.microsoft.com/en-us/azure/data-factory/sap-change-data-capture-introduction-architectu...