How worker nodes get the packages during scale-up?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-01-2024 05:19 AM
Hi,
We are working with one of the repository where we used to download the artifact/python package from that repository using index url in global init script but now the logic is going to be change we need give the cred to download the package and that will be valid for few hours.
So I have the question how the new worker node will get the same package/artifact during scale up it will download using the global init script or it will get it from the driver node so whether it will hit the configured index url or not?
I have second question If I consider that it(new worker node) will execute the global init script and it will again download the package from the configured index url then whether it will get the new credentials from the azure library where the new cred will update or it will execute with the old cred what it got it on driver node during the first execution?
Regards,
Satya
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-09-2025 02:33 AM
Yes, the new worker node will execute the global init script independently when it starts.
It does not get the package from the driver or other existing nodes and will hit the configured index URL directly, and try to download the package on its own. Therefore, every new node joining during autoscaling will invoke the index URL to download the artifact/package.
Regarding the second question, it depends on how your global init script is written.
If you pull the credentials from Databricks secrets every time the script runs, then the new node will get fresh credentials, but if the driver fetches the credentials once and writes them to DBFS or environment variables shared across nodes, then new worker nodes may use stale credentials.
