What are optimized solutions for moving on-premise IBM DB2 CDC data to Databricks Delta table

[MunikrishnaS]

Hi Team,

My requirement is to move build a solution to move zos(db2) CDC data to Delta table on Realtime bases(at least near realtime) , data volume and number of tables are little huge (100 tables) 

I have researched I dont find any inbuild options in ADF or Databricks side, if anybody have good solution to implemented this use case please let me know the approach 

[MunikrishnaS]

Thank you @Retired_mod 

One more question to same subject, is there any inhouse(Azure or Databricks) tools or services are available to read zos(db2) CDC data ? as like Debezium connector? we are using fully Azure databricks platform, trying to figure out any inhouse connectors which can handles reading cdc data from DB2(using Transaction logs information) and loaded it in lakehouse , pls advise if we have any architectural solution already available 

[-werners-]

AFAIK there is no 'standard' tool available.  Debezium is an option.
What CDC system do you use?  Infospere or another one?
Ideally you would be able to write CDC data into Azure Event Hub (or Kafka or ...).
Databricks can connect to this with a streaming query.

[MunikrishnaS]

Thanks @-werners- 

we are in the process of evaluating best CDC process to read data from zos(db2) system and load it in Lakehouse platform, realtime or very near realtime bases (data volume will be huge and around 100+ tables) 

Currently on-Perm system is using IDMS CDC connector to access CDC data from db2 and loading in On-PERM Database.

we are trying to look for optimized or better solution and low latency as possible to data in lakehouse and make available data for users . 

[-werners-]

There are certainly alternatives.  In the partner connect of databricks there are several options, but outside of partner connect there are others too (like Debezuim f.e.).
What the best option is, is not easily answered, as it depends on your environment, budget, requirements etc.
But what is important is that you want the CDC data to be streamed into spark.  That seems to me like a good starting point in selecting a solution (Kafka integration, Event Hub integration).

[MunikrishnaS]

Thank you @-werners-