โ07-01-2022 02:09 AM
Hello,
I have an issue with the import of a custom library, in Azure Databricks.
(roughly) 95% of the times it works fine, but sometimes it fails.
I searched the internet and this community with no luck, so far.
It is a scala library in a scala notebook, and its import command is just like:
import it.foo.bar.MyClass
It works fine, usually; sometimes, with no reason (that I can think of) it fails like this:
command-261888690252589:1: error: not found: value it
If I rerun the very same activity, it works fine.
Do you have any idea why this happens, or how I could investigate it further?
Several such import commands might run at the same time (in parallel, in Azure Data Factory), I don't know if this might cause some trouble.
I could set some retry mechanism, but I would like to understand what exactly is going on here.
Thank you,
Nk
โ09-02-2022 05:43 AM
Hello Jose,
thanks for your concern.
No, no more job failures occurred; when it does, I will gather all possible log messages and share it here, to see if it helps.
I cannot consider the problem solved, until I figure out why this happens.
Kind regards,
Niki
โ07-05-2022 10:08 AM
Hi @Nk 76โ,
Could you share more details please? what us the error stack trace? how to repro this issue? do you have any extra spark settings on your cluster?
โ07-07-2022 10:36 PM
Hi @Nk 76โ, We haven't heard from you on the last response from @Jose Gonzalezโ, and I was checking back to see if you were able to answer his questions. Or else, If you have any solution, please share it with the community as it can be helpful to others. Otherwise, we will respond with more details and try to help.
Also, please don't forget to click on the "Select As Best" button whenever the information provided helps resolve your question.
โ07-13-2022 03:31 AM
Hi @Kaniz Fatmaโ , @Jose Gonzalezโ
the problem has not occurred anymore, for now;
I am afraid I have no further details, there is no stack trace and the issue is not replicable.
Do you have any idea?
Thanks for your time
โ07-13-2022 03:34 AM
Hi @Nk 76โ, Does that mean you don't encounter any problems right now?
โ07-13-2022 04:03 AM
Hi @Kaniz Fatmaโ ,
yes, I mean that I don't have that problem right now, but as I said it happens once in a while, in a (seemingly) random fashion.
So it might happen again tomorrow, or in 1 week, or whenever.
โ07-13-2022 04:45 AM
Hi @Nk 76โ, Thank you for the quick update. We'll look into it and get back soon. Thanks.
โ07-08-2022 12:08 AM
Hello,
I checked but I can see no stack trace. The notebook just says:
command-261888690252589:1: error: not found: value it
import it.foo.bar.aaaa
The error cannot be reproduced, it seems to happen randomly.
I have no extra settings on the Cluster.
I would like to share more details, but I have none, unfortunately.
The Cluster only shows logs of the last few days, and that one is gone, I'm afraid.
Thanks for your time.
โ09-01-2022 04:41 PM
The retention period is 30 days. Do you have any recent job failure?
โ09-02-2022 05:43 AM
Hello Jose,
thanks for your concern.
No, no more job failures occurred; when it does, I will gather all possible log messages and share it here, to see if it helps.
I cannot consider the problem solved, until I figure out why this happens.
Kind regards,
Niki
โ09-02-2022 01:07 PM
Hi @Nk 76โ , Please keep us posted. Thank you for your response.
โ12-05-2022 05:49 PM