cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Delay in files update on filesystem

Jiri_Koutny
New Contributor III

Hi, I noticed that there is quite a significant delay (2 - 10s) between making a change to some file in Repos via Databricks file edit window and propagation of such change to the filesystem. Our engineers and scientists use YAML config files. If they reload a notebook that loads this YAML config too soon, nothing really changes. It is very confusing and causes unexpected situations and errors.

Is it a bug or feature? Thanks

11 REPLIES 11

-werners-
Esteemed Contributor III

Not sure, but what I notice is that the responsiveness of the git interface is not very fast.

F.e. creating a new branch from main can take a while. Perhaps this is a similar thing.

Jiri_Koutny
New Contributor III

Yes, GIT UI is slow too. Based on our research Databricks Repos Files use some network filesystem which is currently quite slow to synchronize changes.

Atanu
Databricks Employee
Databricks Employee

Hi @Jiri Koutny​  are you still observing slowness?

Atanu
Databricks Employee
Databricks Employee

and same has been observed with API too ? https://docs.databricks.com/dev-tools/api/latest/repos.html

jose_gonzalez
Databricks Employee
Databricks Employee

Hi @Jiri Koutny​ ,

Are you still facing slowness issues or not anymore?

Jiri_Koutny
New Contributor III

Hi, yes it is still slow. I had a discussion with Databricks Repos PM and as far as I understand it, there will be some improvements implemented soon.

JoeS
New Contributor III

@Jose Gonzalez​  @Atanu Sarkar​  I'm noticing that the file updates are still slow. I am on 10.4LTS.

When I edit a .py module within a Databricks Repo and import via a notebook in the same folder, the changes are not reflected for many minutes.

Is this expected to be fixed soon?

SCWD
New Contributor III

Just happened to me. Updates to git repo files took ~30 seconds to be reflected in notebook imports.

iulp
New Contributor II

I have the same issue: changes were done > 1 hour ago in the .yaml file, but when I read it in the Notebook I still see the old version. Detach and re-attach and even the restarting of the cluster (DBR ML 14.2) did not help.

DaniyarZ
New Contributor II

There is a trick: if you execute "%sh ls" command, it forces update of filesystem immediately

Irka
New Contributor II

Is there a solution to this?
BTW, the "ls" command trick didn't work for me 😞

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group