NandiniN
Databricks Employee
Databricks Employee

Hi @Nasd_

I believe you are trying to use OSS jars on DBR. (Can infer based on class package)

org.apache.spark.sql.delta.DeltaLog

The error ModuleNotFoundError: No module named 'delta.exceptions.captured'; 'delta.exceptions' is not a package can be seen when installing the open-source delta-spark (or Delta Core) Python package would be a package conflict

Databricks Runtime includes a native version of the Delta Lake Python libraries that are tightly coupled with the binaries on the cluster. When you install the open-source delta-spark package via %pip or as a cluster library, it often overwrites or conflicts with the native Databricks-provided modules, leading to the Python import error because the structure or contents of the installed package do not match what the Databricks environment expects.

Okay, I just see you have the answer on this thread - https://community.databricks.com/t5/data-engineering/accessing-deltalog-and-optimistictransaction-fr... and you have accepted the answer. So, I believe your questions are answered.

Thanks!

View solution in original post