Custom library import fails randomly with error: not found: value it

[nk76]

Hello,

I have an issue with the import of a custom library, in Azure Databricks.

(roughly) 95% of the times it works fine, but sometimes it fails.

I searched the internet and this community with no luck, so far.

It is a scala library in a scala notebook, and its import command is just like:

import it.foo.bar.MyClass

It works fine, usually; sometimes, with no reason (that I can think of) it fails like this:

command-261888690252589:1: error: not found: value it

If I rerun the very same activity, it works fine.

Do you have any idea why this happens, or how I could investigate it further?

Several such import commands might run at the same time (in parallel, in Azure Data Factory), I don't know if this might cause some trouble.

I could set some retry mechanism, but I would like to understand what exactly is going on here.

Thank you,

Nk

[jose_gonzalez]

Hi @Nk 76​,

Could you share more details please? what us the error stack trace? how to repro this issue? do you have any extra spark settings on your cluster?

[Kaniz]

Hi @Nk 76​, We haven't heard from you on the last response from @Jose Gonzalez​, and I was checking back to see if you were able to answer his questions. Or else, If you have any solution, please share it with the community as it can be helpful to others. Otherwise, we will respond with more details and try to help.

Also, please don't forget to click on the "Select As Best" button whenever the information provided helps resolve your question.

[nk76]

Hi @Kaniz Fatma​ , @Jose Gonzalez​ 

the problem has not occurred anymore, for now;

I am afraid I have no further details, there is no stack trace and the issue is not replicable.

Do you have any idea?

Thanks for your time

[Kaniz]

Hi @Nk 76​, Does that mean you don't encounter any problems right now?

[nk76]

Hi @Kaniz Fatma​ ,

yes, I mean that I don't have that problem right now, but as I said it happens once in a while, in a (seemingly) random fashion.

So it might happen again tomorrow, or in 1 week, or whenever.

[Kaniz]

Hi @Nk 76​, Thank you for the quick update. We'll look into it and get back soon. Thanks.

[nk76]

Hello,

I checked but I can see no stack trace. The notebook just says:

command-261888690252589:1: error: not found: value it
import it.foo.bar.aaaa

The error cannot be reproduced, it seems to happen randomly.

I have no extra settings on the Cluster.

I would like to share more details, but I have none, unfortunately.

The Cluster only shows logs of the last few days, and that one is gone, I'm afraid.

Thanks for your time.

[jose_gonzalez]

The retention period is 30 days. Do you have any recent job failure?

[nk76 · Accepted solution]

Hello Jose,

thanks for your concern.

No, no more job failures occurred; when it does, I will gather all possible log messages and share it here, to see if it helps.

I cannot consider the problem solved, until I figure out why this happens.

Kind regards,

Niki

[Kaniz]

Hi @Nk 76​ , Please keep us posted. Thank you for your response.

[Naskar]

Even I also encountered the same error. While Importing a file getting an error as "Import failed with error: Could not deserialize: Exceeded 16777216 bytes (current = 16778609)"