Long run time with %run command

[cmilligan]

My team has started to see long run times on cells when using the %run commands to run another notebook. The notebook that we are calling with %run only contains variable setting, defining functions, and library imports. In some cases I have seen in excess of 10+ minutes but this isn't behavior I would expect without actually running anything.

Has anyone else run into this and how have you resolved it?

[cmilligan]

Thank you for the reply @Retired_mod ,

I don't think the issue is with the performance of the notebook that we're calling with %run. The only things in this notebook are re-usable python functions and simple variable setting (text strings, passwords, static lists, etc.). A majority of the time when running this notebook using %run it takes a couple of seconds or less. Lately it seems that when there is high demand on the cluster it can take upwards of 10 minutes or we will just experience connection timeout failures. This tends to happen more on this command when running as part of a larger job or another job is running on the cluster. 

[cmilligan]

@Retired_mod  it seems like the issue is accessing secrets in the scope. I was testing with a user that doesn't have access to the secret scope which is one of the first commands in the notebook. I would expect it to fail quickly since they don't have access but it still continues to run for a long time.

[Debayan]

Hi @cmilligan , Can you try this with another user? Also, with a different notebook and cluster? What is the DBR version now running? 

[cmilligan]

Hi @Debayan,

I've tried this against multiple users and notebooks. We've also used multiple clusters one with 10.4 LTS and the other with 13.3 LTS. The issue is still happening

[Debayan]

Hi, Do you see anything suspicious in the log4j section of the cluster driver logs? 

[cmilligan]

@Debayan I'm not really sure, I haven't read the logs before, and looking through it I'm not sure if there is something that should stand out or not