โ07-01-2022 02:09 AM
Hello,
I have an issue with the import of a custom library, in Azure Databricks.
(roughly) 95% of the times it works fine, but sometimes it fails.
I searched the internet and this community with no luck, so far.
It is a scala library in a scala notebook, and its import command is just like:
import it.foo.bar.MyClass
It works fine, usually; sometimes, with no reason (that I can think of) it fails like this:
command-261888690252589:1: error: not found: value it
If I rerun the very same activity, it works fine.
Do you have any idea why this happens, or how I could investigate it further?
Several such import commands might run at the same time (in parallel, in Azure Data Factory), I don't know if this might cause some trouble.
I could set some retry mechanism, but I would like to understand what exactly is going on here.
Thank you,
Nk
โ09-02-2022 05:43 AM
Hello Jose,
thanks for your concern.
No, no more job failures occurred; when it does, I will gather all possible log messages and share it here, to see if it helps.
I cannot consider the problem solved, until I figure out why this happens.
Kind regards,
Niki
โ07-05-2022 10:08 AM
Hi @Nk 76โ,
Could you share more details please? what us the error stack trace? how to repro this issue? do you have any extra spark settings on your cluster?
โ07-13-2022 03:31 AM
Hi @Kaniz Fatmaโ , @Jose Gonzalezโ
the problem has not occurred anymore, for now;
I am afraid I have no further details, there is no stack trace and the issue is not replicable.
Do you have any idea?
Thanks for your time
โ07-13-2022 04:03 AM
Hi @Kaniz Fatmaโ ,
yes, I mean that I don't have that problem right now, but as I said it happens once in a while, in a (seemingly) random fashion.
So it might happen again tomorrow, or in 1 week, or whenever.
โ07-08-2022 12:08 AM
Hello,
I checked but I can see no stack trace. The notebook just says:
command-261888690252589:1: error: not found: value it
import it.foo.bar.aaaa
The error cannot be reproduced, it seems to happen randomly.
I have no extra settings on the Cluster.
I would like to share more details, but I have none, unfortunately.
The Cluster only shows logs of the last few days, and that one is gone, I'm afraid.
Thanks for your time.
โ09-01-2022 04:41 PM
The retention period is 30 days. Do you have any recent job failure?
โ09-02-2022 05:43 AM
Hello Jose,
thanks for your concern.
No, no more job failures occurred; when it does, I will gather all possible log messages and share it here, to see if it helps.
I cannot consider the problem solved, until I figure out why this happens.
Kind regards,
Niki
โ12-05-2022 05:49 PM
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโt want to miss the chance to attend and share knowledge.
If there isnโt a group near you, start one and help create a community that brings people together.
Request a New Group