cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Custom library import fails randomly with error: not found: value it

nk76
New Contributor III

Hello,

I have an issue with the import of a custom library, in Azure Databricks.

(roughly) 95% of the times it works fine, but sometimes it fails.

I searched the internet and this community with no luck, so far.

It is a scala library in a scala notebook, and its import command is just like:

import it.foo.bar.MyClass

It works fine, usually; sometimes, with no reason (that I can think of) it fails like this:

command-261888690252589:1: error: not found: value it

If I rerun the very same activity, it works fine.

Do you have any idea why this happens, or how I could investigate it further?

Several such import commands might run at the same time (in parallel, in Azure Data Factory), I don't know if this might cause some trouble.

I could set some retry mechanism, but I would like to understand what exactly is going on here.

Thank you,

Nk

1 ACCEPTED SOLUTION

Accepted Solutions

nk76
New Contributor III

Hello Jose,

thanks for your concern.

No, no more job failures occurred; when it does, I will gather all possible log messages and share it here, to see if it helps.

I cannot consider the problem solved, until I figure out why this happens.

Kind regards,

Niki

View solution in original post

7 REPLIES 7

jose_gonzalez
Moderator
Moderator

Hi @Nk 76​,

Could you share more details please? what us the error stack trace? how to repro this issue? do you have any extra spark settings on your cluster?

nk76
New Contributor III

Hi @Kaniz Fatma​ , @Jose Gonzalez​ 

the problem has not occurred anymore, for now;

I am afraid I have no further details, there is no stack trace and the issue is not replicable.

Do you have any idea?

Thanks for your time

nk76
New Contributor III

Hi @Kaniz Fatma​ ,

yes, I mean that I don't have that problem right now, but as I said it happens once in a while, in a (seemingly) random fashion.

So it might happen again tomorrow, or in 1 week, or whenever.

nk76
New Contributor III

Hello,

I checked but I can see no stack trace. The notebook just says:

command-261888690252589:1: error: not found: value it
import it.foo.bar.aaaa

The error cannot be reproduced, it seems to happen randomly.

I have no extra settings on the Cluster.

I would like to share more details, but I have none, unfortunately.

The Cluster only shows logs of the last few days, and that one is gone, I'm afraid.

Thanks for your time.

The retention period is 30 days. Do you have any recent job failure?

nk76
New Contributor III

Hello Jose,

thanks for your concern.

No, no more job failures occurred; when it does, I will gather all possible log messages and share it here, to see if it helps.

I cannot consider the problem solved, until I figure out why this happens.

Kind regards,

Niki

Naskar
New Contributor II

Even I also encountered the same error. While Importing a file getting an error as "Import failed with error: Could not deserialize: Exceeded 16777216 bytes (current = 16778609)"

image

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group