topic Custom library import fails randomly with error: not found: value it in Data Engineering

Custom library import fails randomly with error: not found: value it

nk76 — Fri, 01 Jul 2022 09:09:09 GMT

Hello,

I have an issue with the import of a custom library, in Azure Databricks.

(roughly) 95% of the times it works fine, but sometimes it fails.

I searched the internet and this community with no luck, so far.

It is a scala library in a scala notebook, and its import command is just like:

import it.foo.bar.MyClass

It works fine, usually; sometimes, with no reason (that I can think of) it fails like this:

command-261888690252589:1: error: not found: value it

If I rerun the very same activity, it works fine.

Do you have any idea why this happens, or how I could investigate it further?

Several such import commands might run at the same time (in parallel, in Azure Data Factory), I don't know if this might cause some trouble.

I could set some retry mechanism, but I would like to understand what exactly is going on here.

Thank you,

Re: Custom library import fails randomly with error: not found: value it

jose_gonzalez — Tue, 05 Jul 2022 17:08:40 GMT

Hi @Nk 76,

Could you share more details please? what us the error stack trace? how to repro this issue? do you have any extra spark settings on your cluster?

Re: Custom library import fails randomly with error: not found: value it

nk76 — Fri, 08 Jul 2022 07:08:58 GMT

Hello,

I checked but I can see no stack trace. The notebook just says:

command-261888690252589:1: error: not found: value it
import it.foo.bar.aaaa

The error cannot be reproduced, it seems to happen randomly.

I have no extra settings on the Cluster.

I would like to share more details, but I have none, unfortunately.

The Cluster only shows logs of the last few days, and that one is gone, I'm afraid.

Thanks for your time.

Re: Custom library import fails randomly with error: not found: value it

nk76 — Wed, 13 Jul 2022 10:31:10 GMT

Hi @Kaniz Fatma , @Jose Gonzalez

the problem has not occurred anymore, for now;

I am afraid I have no further details, there is no stack trace and the issue is not replicable.

Do you have any idea?

Thanks for your time

Re: Custom library import fails randomly with error: not found: value it

nk76 — Wed, 13 Jul 2022 11:03:17 GMT

Hi @Kaniz Fatma ,

yes, I mean that I don't have that problem right now, but as I said it happens once in a while, in a (seemingly) random fashion.

So it might happen again tomorrow, or in 1 week, or whenever.

Re: Custom library import fails randomly with error: not found: value it

jose_gonzalez — Thu, 01 Sep 2022 23:41:39 GMT

The retention period is 30 days. Do you have any recent job failure?

Re: Custom library import fails randomly with error: not found: value it

nk76 — Fri, 02 Sep 2022 12:43:54 GMT

Hello Jose,

thanks for your concern.

No, no more job failures occurred; when it does, I will gather all possible log messages and share it here, to see if it helps.

I cannot consider the problem solved, until I figure out why this happens.

Kind regards,

Niki

Re: Custom library import fails randomly with error: not found: value it

Naskar — Tue, 06 Dec 2022 01:49:46 GMT

Even I also encountered the same error. While Importing a file getting an error as "Import failed with error: Could not deserialize: Exceeded 16777216 bytes (current = 16778609)"