How to specify init file path

MikeGo — Wed, 19 Feb 2025 01:32:34 GMT

Hi team,

I want to create a job and install some libs to job cluster.

1. For job cluster, understand we can specify libraries under task, but if I want to install the lib to the whole cluster seems the only way is to use init script right?

2. In my env, I want to put the init script directly with source code instead of uploading to DBFS first. I need to specify an absolute path for it like

/Workspace/Users/.../.bundle/.../files/src/bash/xxx.sh

Is there a way to dynamically specify the user ID as it can be different in different env?

Thanks

Re: How to specify init file path

kamal_ch — Tue, 13 May 2025 02:28:53 GMT

There is no direct method to dynamically specify the user ID in this path across environments.

If dynamic determination of the user ID per environment is essential, you may need additional scripting or automation to set the correct path before executing the init configuration. Alternatively, consider structuring paths and environments consistently to avoid hardcoding specific user IDs

topic Re: How to specify init file path in Data Engineering

How to specify init file path

Re: How to specify init file path