topic Re: TypeError: 'JavaPackage' object is not callable in Data Engineering

TypeError: 'JavaPackage' object is not callable

chandan_a_v — Thu, 02 Jun 2022 08:49:34 GMT

Re: TypeError: 'JavaPackage' object is not callable

Prabakar — Thu, 02 Jun 2022 10:07:37 GMT

@Chandan Angadi, which library are you using? Are you installing using the libraries tab or are you using an init script to install the library. Is this happening always or intermittent?

Re: TypeError: 'JavaPackage' object is not callable

chandan_a_v — Thu, 02 Jun 2022 11:24:27 GMT

Hi @Prabakar Ammeappin ,

I am using set of libraries, which I am installing under a notebook. Not in the cluster config.

python>=3.7,<3.8

findspark=1.3.0

openpyxl=3.0.7

pyarrow=0.14.0

smart_open=5.2.1

xlrd=2.0.1

conda-pack=0.6.0

tqdm=4.62.2

tsfresh=0.17.0

scikit-learn=0.24.2

pip=21.2.4

git=2.33.0

pandas=1.2.5

fsspec=2021.7.0

mlflow==1.20.1

pydeequ==0.1.5

s3fs==2021.8.0

botocore==1.20.106

boto3==1.17.106

pyspark==2.4.7

torch-tb-profiler==0.4.0

pytorch-forecasting==0.9.2

pytorch-lightning==1.4.5

tensorboard==2.8.0

ipyaggrid==0.2.1

jupyter-contrib-nbextensions==0.5.1

fastparquet==0.7.1

plotnine==0.8.0

tslearn==0.5.2

fastdtw==0.3.4

dtaidistance==2.3.2

catboost==1.0.3

shap==0.40.0

k-means-constrained==0.6.0

pdf2image==1.16.0

kaleido==0.2.1

pickle5==0.0.12

Thanks,

Chandan

Re: TypeError: 'JavaPackage' object is not callable

Prabakar — Thu, 02 Jun 2022 15:24:00 GMT

By any chance, was the cluster restarted after installing the libraries or was it detached and reattached from/to the notebook? Notebook-scoped libraries do not persist across sessions. You must reinstall notebook-scoped libraries at the beginning of each session, or whenever the notebook is detached from a cluster.

Re: TypeError: 'JavaPackage' object is not callable

chandan_a_v — Fri, 03 Jun 2022 04:52:30 GMT

HI @Prabakar Ammeappin ,

Yes, I agree notebook scoped lib do not persist across sessions. I am installing the required lib in the first cell of the notebook and afterward rest of the code will be executed. The notebook is not detached from the cluster.

Re: TypeError: 'JavaPackage' object is not callable

Prabakar — Fri, 03 Jun 2022 08:44:08 GMT

We need to debug the logs to understand this further. Is it possible for you to export and share a notebook with minimal repro steps? It will help us to reproduce this in-house and check the logs. Or If you have a support contract, it would be better to raise a support ticket.

Re: TypeError: 'JavaPackage' object is not callable

chandan_a_v — Thu, 09 Jun 2022 11:13:44 GMT

Hi @Kaniz Fatma ,

Sorry for the late response, Installing deequ-2.0.1-spark-3.2 on cluster solves the issue.

Re: TypeError: 'JavaPackage' object is not callable

chandan_a_v — Thu, 09 Jun 2022 11:22:03 GMT

The spark version of Jar file must match with Cluster Spark version