Import module never ends on random branches

[Valentin14]

Hello,

Since a week ago, our notebook are stuck in running on the firsts cells which import python module from our github repository which is cloned in databricks.

The cells stays in running state and when we try to manually cancel the jobs in databricks, it remains in cancelling and never end until the cluster is terminated.

What is really weird, is that we have the issue only on specific branches, and if we recreate the branch there is no more issue...

When the notebooks are submitted from adf, i got the following error(when it fails, some notebooks run ok, others doesn't) :

OSError: [Errno 5] Input/output error: '/Workspace/Repos/datafactory-view-develop/***/src/curated/business'

The workspaces are clones from github repo, so I don't understand why i have issues when loading module from our workspaces, when same module is loaded in another notebook without issue...

I have close to no logs describing the issue in the cluster logs so it's really hard to identify the cause of this issue..

The cluster runs in Azure environment.

Does anyone already encountered this kind of issue ?

[Debayan]

Hi @Valentin Rosca​, Without the logs it will be very hard to determine the cause of the issue. As the first step could you please raise a support case to Azure so that they can do the primary investigation on this and reach out to us if anything is needed.

Please tag @Debayan​ with your next comment so that I will get notified. Thank you!

[Anonymous]

Hi @Valentin LEBOSSE​ 

Thank you for posting your question in our community! We are happy to assist you.

To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your question?

This will also help other community members who may have similar questions in the future. Thank you for your participation and let us know if you need any further assistance! 

[mgradowski]

Error from FUSE process during `ls /Workspace`:

2024/06/26 11:15:39.286089 workspace.WARNING [CP] ResponseLogHook: databricks api https://westeurope-c2.azuredatabricks.net/api/2.0/workspace-files/list-files: 504 Gateway Timeout. body:{"error_code":"TEMPORARILY_UNAVAILABLE","message":"The service at /api/2.0/workspace-files/list-files is taking too long to process your request. Please try again later or try a faster operation. [TraceId: 00-0a794387a987fc4de1cb8a6b70214774-97e61adb2ab644d5-00]"}

[Kaniz_Fatma]

Hi @mgradowski,

• This error can occur if the mount directory is left mounted due to a previous crash of your filesystem.
• Ensure that you have the necessary permissions to access the FUSE directory.

[timo199]

How do you determine what permissions are required to access Fuse?

Please answer in detail

[timo199]

@Kaniz_Fatma