07-28-2023 01:57 AM
Hello guys. I am having the following annoying issue with DataBricks job executed from ADF. I have stopped all the logs, deleted all the print statements and display. And I am still getting this error. I don't understand what might cause it. I am attaching a screenshot from the driver log, which is showing that stdout log file is no more than 10 kb
07-28-2023 04:36 AM
Forgot to mention it is triggered only from ADF, in DataBricks the execution goes without any troubles
08-01-2023 03:23 AM
Do you happen to call other notebooks within that notebook? Or have you entered a lot of markdown?
08-14-2023 12:49 PM
None of those
08-20-2023 11:05 PM
@Chalki Glad to hear it is working fine. Please let us know if any other help is needed.
08-01-2023 04:52 AM
Hi @Chalki
Is the output of the notebook being passed into any other ADF activity ? If yes, check that output.
Also, can you give further details like the configuration you are using in ADB and in ADF for compute ?
08-14-2023 12:48 PM - edited 08-14-2023 12:49 PM
It is, but it is a short string, like a "success" message
08-14-2023 06:30 AM
Hi, Could you please upload the log4j logs here if thats okay? Also, please tag @Debayan with your next comment, which will notify me.
08-14-2023 12:45 PM - edited 08-14-2023 12:50 PM
Apologies for the late reply. I will try to upload the logs,cause it's been some time.
Interesting is how I solve it. I put all the code in a single cell and It is running since then without issues. Although it was just few cells with python code. I even increased the logging and I have no troubles for the time being
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.
If there isn’t a group near you, start one and help create a community that brings people together.
Request a New Group