cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Delay in files update on filesystem

Jiri_Koutny
New Contributor III

Hi, I noticed that there is quite a significant delay (2 - 10s) between making a change to some file in Repos via Databricks file edit window and propagation of such change to the filesystem. Our engineers and scientists use YAML config files. If they reload a notebook that loads this YAML config too soon, nothing really changes. It is very confusing and causes unexpected situations and errors.

Is it a bug or feature? Thanks

10 REPLIES 10

-werners-
Esteemed Contributor III

Not sure, but what I notice is that the responsiveness of the git interface is not very fast.

F.e. creating a new branch from main can take a while. Perhaps this is a similar thing.

Jiri_Koutny
New Contributor III

Yes, GIT UI is slow too. Based on our research Databricks Repos Files use some network filesystem which is currently quite slow to synchronize changes.

Atanu
Esteemed Contributor
Esteemed Contributor

Hi @Jiri Koutny​  are you still observing slowness?

Atanu
Esteemed Contributor
Esteemed Contributor

and same has been observed with API too ? https://docs.databricks.com/dev-tools/api/latest/repos.html

jose_gonzalez
Moderator
Moderator

Hi @Jiri Koutny​ ,

Are you still facing slowness issues or not anymore?

Jiri_Koutny
New Contributor III

Hi, yes it is still slow. I had a discussion with Databricks Repos PM and as far as I understand it, there will be some improvements implemented soon.

JoeS
New Contributor III

@Jose Gonzalez​  @Atanu Sarkar​  I'm noticing that the file updates are still slow. I am on 10.4LTS.

When I edit a .py module within a Databricks Repo and import via a notebook in the same folder, the changes are not reflected for many minutes.

Is this expected to be fixed soon?

SCWD
New Contributor III

Just happened to me. Updates to git repo files took ~30 seconds to be reflected in notebook imports.

iulp
New Contributor II

I have the same issue: changes were done > 1 hour ago in the .yaml file, but when I read it in the Notebook I still see the old version. Detach and re-attach and even the restarting of the cluster (DBR ML 14.2) did not help.

DaniyarZ
New Contributor II

There is a trick: if you execute "%sh ls" command, it forces update of filesystem immediately

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!