Hi @Anil_M, Absolutely, the error message you're facing, "TypeError: 'JavaPackage' object is not callable," usually arises when there's a problem with how you're calling a method or function in your code. Let's dive into troubleshooting this issue together: Firstly, it's crucial to check if you've installed all the required dependencies. BERT embeddings might need specific libraries or packages, so ensure they've been correctly installed. Secondly, please verify that you possess the necessary versions of Spark and PySpark. As you mentioned, you're using Spark version 3.3.0, which should be compatible with BERT embeddings.
To ensure successful model retrieval, double-check that your specified model path accurately leads to the stored BERT model. Verify that your Databricks cluster has proper access and authorization to the model files. Additionally, be mindful of library versions and their potential for compatibility conflicts. Validate that PySpark, BERT, and any relevant libraries are all compatible with one another. In the case of custom libraries, be certain that they are properly installed and accessible within your notebook.
When setting up your cluster, it's important to keep in mind that libraries installed within a notebook are not permanent and will need to be reinstalled at the start of each session or when the notebook is connected to a cluster. Before troubleshooting, ensure that the cluster has not been restarted or detached since installing the necessary libraries.
To possibly solve this issue, users have shared their success with installing deequ-2.0.1-spark-3.2 on their cluster. It could be worth trying out this approach. Keep in mind that troubleshooting specific problems often requires delving into logs and carefully analyzing your code's context. Don't hesitate to provide more details or ask any additional questions โ let's work together to get to the bottom of this! ๐
