Delay in files update on filesystem

[Jiri_Koutny]

Hi, I noticed that there is quite a significant delay (2 - 10s) between making a change to some file in Repos via Databricks file edit window and propagation of such change to the filesystem. Our engineers and scientists use YAML config files. If they reload a notebook that loads this YAML config too soon, nothing really changes. It is very confusing and causes unexpected situations and errors.

Is it a bug or feature? Thanks

[-werners-]

Not sure, but what I notice is that the responsiveness of the git interface is not very fast.

F.e. creating a new branch from main can take a while. Perhaps this is a similar thing.

[Jiri_Koutny]

Yes, GIT UI is slow too. Based on our research Databricks Repos Files use some network filesystem which is currently quite slow to synchronize changes.

[Atanu]

Hi @Jiri Koutny​  are you still observing slowness?

[Atanu]

and same has been observed with API too ? https://docs.databricks.com/dev-tools/api/latest/repos.html

[jose_gonzalez]

Hi @Jiri Koutny​ ,

Are you still facing slowness issues or not anymore?

[Jiri_Koutny]

Hi, yes it is still slow. I had a discussion with Databricks Repos PM and as far as I understand it, there will be some improvements implemented soon.

[JoeS]

@Jose Gonzalez​  @Atanu Sarkar​  I'm noticing that the file updates are still slow. I am on 10.4LTS.

When I edit a .py module within a Databricks Repo and import via a notebook in the same folder, the changes are not reflected for many minutes.

Is this expected to be fixed soon?

[SCWD]

Just happened to me. Updates to git repo files took ~30 seconds to be reflected in notebook imports.

[iulp]

I have the same issue: changes were done > 1 hour ago in the .yaml file, but when I read it in the Notebook I still see the old version. Detach and re-attach and even the restarting of the cluster (DBR ML 14.2) did not help.

[DaniyarZ]

There is a trick: if you execute "%sh ls" command, it forces update of filesystem immediately