What are optimized solutions for moving on-premise IBM DB2 CDC data to Databricks Delta table

[MunikrishnaS]

Hi Team,

My requirement is to move build a solution to move zos(db2) CDC data to Delta table on Realtime bases(at least near realtime) , data volume and number of tables are little huge (100 tables) 

I have researched I dont find any inbuild options in ADF or Databricks side, if anybody have good solution to implemented this use case please let me know the approach 

[Kaniz]

Hi @MunikrishnaS, To implement a solution for moving z/OS (DB2) Change Data Capture (CDC) data to Delta tables on a near real-time basis, you can consider the following approach:

Databricks Delta Live Tables (DLT): Databricks has introduced a feature in Delta Live Tables that si.... DLT allows users to ingest CDC data seamlessly using SQL and Python. The new APPLY CHANGES INTO operation in DLT pipelines automatically and seamlessly handles out-of-or....

MERGE INTO command in Delta Lake on Databricks enables customers to upsert and delete records in the....

[MunikrishnaS]

Thank you @Kaniz 

One more question to same subject, is there any inhouse(Azure or Databricks) tools or services are available to read zos(db2) CDC data ? as like Debezium connector? we are using fully Azure databricks platform, trying to figure out any inhouse connectors which can handles reading cdc data from DB2(using Transaction logs information) and loaded it in lakehouse , pls advise if we have any architectural solution already available 

[-werners-]

AFAIK there is no 'standard' tool available.  Debezium is an option.
What CDC system do you use?  Infospere or another one?
Ideally you would be able to write CDC data into Azure Event Hub (or Kafka or ...).
Databricks can connect to this with a streaming query.

[MunikrishnaS]

Thanks @-werners- 

we are in the process of evaluating best CDC process to read data from zos(db2) system and load it in Lakehouse platform, realtime or very near realtime bases (data volume will be huge and around 100+ tables) 

Currently on-Perm system is using IDMS CDC connector to access CDC data from db2 and loading in On-PERM Database.

we are trying to look for optimized or better solution and low latency as possible to data in lakehouse and make available data for users . 

[-werners-]

There are certainly alternatives.  In the partner connect of databricks there are several options, but outside of partner connect there are others too (like Debezuim f.e.).
What the best option is, is not easily answered, as it depends on your environment, budget, requirements etc.
But what is important is that you want the CDC data to be streamed into spark.  That seems to me like a good starting point in selecting a solution (Kafka integration, Event Hub integration).

[MunikrishnaS]

Thank you @-werners- 

[Kaniz]

Thank you for posting your question in our community! We are happy to assist you.

To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your question?

This will also help other community members who may have similar questions in the future. Thank you for your participation and let us know if you need any further assistance!