topic Re: Delay in files update on filesystem in Data Engineering

Delay in files update on filesystem

Jiri_Koutny — Thu, 25 Nov 2021 12:47:28 GMT

Hi, I noticed that there is quite a significant delay (2 - 10s) between making a change to some file in Repos via Databricks file edit window and propagation of such change to the filesystem. Our engineers and scientists use YAML config files. If they reload a notebook that loads this YAML config too soon, nothing really changes. It is very confusing and causes unexpected situations and errors.

Is it a bug or feature? Thanks

Re: Delay in files update on filesystem

-werners- — Thu, 25 Nov 2021 12:55:42 GMT

Not sure, but what I notice is that the responsiveness of the git interface is not very fast.

F.e. creating a new branch from main can take a while. Perhaps this is a similar thing.

Re: Delay in files update on filesystem

Jiri_Koutny — Fri, 26 Nov 2021 08:03:15 GMT

Yes, GIT UI is slow too. Based on our research Databricks Repos Files use some network filesystem which is currently quite slow to synchronize changes.

Re: Delay in files update on filesystem

Atanu — Tue, 07 Dec 2021 05:45:04 GMT

Hi @Jiri Koutny are you still observing slowness?

Re: Delay in files update on filesystem

Atanu — Tue, 07 Dec 2021 05:45:40 GMT

and same has been observed with API too ? https://docs.databricks.com/dev-tools/api/latest/repos.html

Re: Delay in files update on filesystem

jose_gonzalez — Tue, 14 Dec 2021 00:56:35 GMT

Hi @Jiri Koutny ,

Are you still facing slowness issues or not anymore?

Re: Delay in files update on filesystem

Jiri_Koutny — Thu, 16 Dec 2021 14:03:34 GMT

Hi, yes it is still slow. I had a discussion with Databricks Repos PM and as far as I understand it, there will be some improvements implemented soon.

Re: Delay in files update on filesystem

JoeS — Thu, 05 May 2022 17:47:28 GMT

@Jose Gonzalez @Atanu Sarkar I'm noticing that the file updates are still slow. I am on 10.4LTS.

When I edit a .py module within a Databricks Repo and import via a notebook in the same folder, the changes are not reflected for many minutes.

Is this expected to be fixed soon?

Re: Delay in files update on filesystem

SCWD — Tue, 14 Mar 2023 20:41:23 GMT

Just happened to me. Updates to git repo files took ~30 seconds to be reflected in notebook imports.

Re: Delay in files update on filesystem

iulp — Fri, 12 Jan 2024 09:45:16 GMT

I have the same issue: changes were done > 1 hour ago in the .yaml file, but when I read it in the Notebook I still see the old version. Detach and re-attach and even the restarting of the cluster (DBR ML 14.2) did not help.

Re: Delay in files update on filesystem

DaniyarZ — Tue, 02 Apr 2024 10:17:35 GMT

There is a trick: if you execute "%sh ls" command, it forces update of filesystem immediately

Re: Delay in files update on filesystem

Irka — Tue, 25 Jun 2024 16:15:41 GMT

Is there a solution to this?
BTW, the "ls" command trick didn't work for me 😞

Re: Delay in files update on filesystem

datadrivenangel — Tue, 09 Sep 2025 14:33:01 GMT

Similar issue where opening up the git interface can take 30+ seconds to show changes.