VSCode Databricks Extension Performance

[Ricklen]

Hello Everyone!

I've been using the Databricks extension in VSCode for a while know and I'm syncing my repository to my Databricks workspace. In the beginning syncing files to my workspace was basically instant. But now it is starting to take a lot of time to sync files (2 to 3 minutes).

I have the feeling there is some cache buildup in VSCode, I already tried to clear the json logging files from Databricks, but this does not help. Also tried syncing manually with a different interval but it doesn't work either.

databricks sync ./my-folder/ /Users/someone@example.com/ --watch --interval 5s

Anyone having the same issue or have ideas in order to get back to the usual upload time?

[Kaniz_Fatma · Accepted solution]

Hi @Ricklen, Please ensure that you have the latest version of the Databricks extension installed. Sometimes, performance issues are resolved in newer versions.

[Ricklen]

Hi Kaniz,

I manage to fix this by reducing the size of the folder that I'm syncing towards Databricks. I assume when using the option '--watch' it will get somewhat slower when there are a lot of directories, subdirectories and files.

Thank you!