Custom library import fails randomly with error: not found: value it

[nk76]

Hello,

I have an issue with the import of a custom library, in Azure Databricks.

(roughly) 95% of the times it works fine, but sometimes it fails.

I searched the internet and this community with no luck, so far.

It is a scala library in a scala notebook, and its import command is just like:

import it.foo.bar.MyClass

It works fine, usually; sometimes, with no reason (that I can think of) it fails like this:

command-261888690252589:1: error: not found: value it

If I rerun the very same activity, it works fine.

Do you have any idea why this happens, or how I could investigate it further?

Several such import commands might run at the same time (in parallel, in Azure Data Factory), I don't know if this might cause some trouble.

I could set some retry mechanism, but I would like to understand what exactly is going on here.

Thank you,

Nk

[jose_gonzalez]

Hi @Nk 76​,

Could you share more details please? what us the error stack trace? how to repro this issue? do you have any extra spark settings on your cluster?

[nk76]

Hi @Kaniz Fatma​ , @Jose Gonzalez​ 

the problem has not occurred anymore, for now;

I am afraid I have no further details, there is no stack trace and the issue is not replicable.

Do you have any idea?

Thanks for your time

[nk76]

Hi @Kaniz Fatma​ ,

yes, I mean that I don't have that problem right now, but as I said it happens once in a while, in a (seemingly) random fashion.

So it might happen again tomorrow, or in 1 week, or whenever.

[nk76]

Hello,

I checked but I can see no stack trace. The notebook just says:

command-261888690252589:1: error: not found: value it
import it.foo.bar.aaaa

The error cannot be reproduced, it seems to happen randomly.

I have no extra settings on the Cluster.

I would like to share more details, but I have none, unfortunately.

The Cluster only shows logs of the last few days, and that one is gone, I'm afraid.

Thanks for your time.

[jose_gonzalez]

The retention period is 30 days. Do you have any recent job failure?

[nk76 · Accepted solution]

Hello Jose,

thanks for your concern.

No, no more job failures occurred; when it does, I will gather all possible log messages and share it here, to see if it helps.

I cannot consider the problem solved, until I figure out why this happens.

Kind regards,

Niki

[Naskar]

Even I also encountered the same error. While Importing a file getting an error as "Import failed with error: Could not deserialize: Exceeded 16777216 bytes (current = 16778609)"