DLT run filas with "com.databricks.cdc.spark.DebeziumJDBCMicroBatchProvider not found"

[smedegaard]

I've created a streaming live table from a foreign catalog. When I run the DLT pipeline it fils with "com.databricks.cdc.spark.DebeziumJDBCMicroBatchProvider not found".

I haven't seen any documentation that suggests I need to install Debezium manually for DLT.
What am I missing?
```
CREATE OR REFRESH STREAMING LIVE TABLE temp_X_1
AS
SELECT
hourvalues.Value
, hourvalues.DateTime
, reduction.EName
, object_descriptions.Description
, units.UnitTxt
FROM
STREAM(`raw-data`.dbo.reduction) AS reduction
INNER JOIN
STREAM(`raw-data`.dbo.hourvalues) AS hourvalues
ON hourvalues.ReportIx = reduction.ReportIx
INNER JOIN
STREAM(`raw-data`.dbo.tbdme_objectdesc) AS object_descriptions
ON object_descriptions.EName = reduction.EName
INNER JOIN
STREAM(`raw-data`.dbo.tbdme_objectUnit) AS units
ON units.EName = reduction.EName
WHERE
reduction.EName LIKE 'XXX%'
```

[Kaniz]

Hi @smedegaard, The error message you’re encountering, “com.databricks.cdc.spark.DebeziumJDBCMicroBatchProvider not found,” indicates that the specified class is not available in your classpath.

To address this issue, follow these steps:

1. Verify Dependencies:

   • Make sure that the required library or package containing the missing class (com.databricks.cdc.spark.DebeziumJDBCMicroBatchProvider) is correctly included in your environment.
   • If you’re using Maven or SBT, ensure that the appropriate dependencies are added to your project configuration.

2. Check Databricks Documentation:

   • While Debezium is commonly used for change data capture (CDC), Databricks Delta Lake (DLT) doesn’t necessarily require manual installation of Debezium.

3. Classpath and Environment:

   • Ensure that the classpath for your DLT pipeline includes the necessary JAR files or packages containing Debezium-related classes.
   • If you’re running your DLT pipeline on Databricks, check the cluster configuration and verify that the required libraries are attached to the cluster.

4. Restart the Cluster:

   • Sometimes, changes to dependencies or configurations require a cluster restart. Try restarting your Databricks cluster after making any adjustments.

Remember to adapt these steps to your specific environment and setup. If you encounter any further issues, consider reaching out for additional assistance¹².

Good luck with resolving the issue! 😊

To ensure we provide you with the best support, could you please take a moment to review the response and choose the one that best answers your question? Your feedback not only helps us assist you better but also benefits other community members who may have similar questions in the future.

If you found the answer helpful, consider giving it a kudo. If the response fully addresses your question, please mark it as the accepted solution. This will help us close the thread and ensure your question is resolved.

We appreciate your participation and are here to assist you further if you need it!

[smedegaard]

This answer is very similar to what the LLMs told me. And unfortunately not helpfuL in solving my problems. 

[Kaniz]

Hi @smedegaard, If you’ve made changes to dependencies or configurations, restart your Databricks cluster to apply the changes.