cancel
Showing results for 
Search instead for 
Did you mean: 
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Showing results for 
Search instead for 
Did you mean: 

How worker nodes get the packages during scale-up?

satycse06
New Contributor

Hi,

We are working with one of the repository where we used to download the artifact/python package from that repository using index url in global init script but now the logic is going to be change we need give the cred to download the package and that will be valid for few hours.

So I have the question how the new worker node will get the same package/artifact during scale up it will download using the global init script or it will get it from the driver node so whether it will hit the configured index url or not?

I have second question If I consider that it(new worker node)  will execute the global init script and it will again download the package from the configured index url  then whether it will get the new credentials from the azure library where the new cred will update or it will execute with the old cred what it got it on driver node during the first execution?

Regards,

Satya

0 REPLIES 0

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group