Job Preview in ADF
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-23-2025 04:39 AM
I do have one Spark Job that is triggered via ADF as a usual "Python" activity. Now wanted to move to Job which is under Preview.
Normally under linked service level i do have spark config and environment that is needed for the execution of this script. I do not find the way how to bypass this spark envs and config to the job cluster i do have created for my new Job Activity.
Anyone faced this before ?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-23-2025 04:52 AM
Hi @fjrodriguez
my understanding is You've already created a cluster for Your job. If that's the case, You can put that spark configuration and env variables directly in the cluster Your job is using.
If for some reason thats not possible, then You can still use 'Additional cluster settings' in ADF Linked Service, providing Cluster spark conf and Cluster Spark environment variables.
Best,
Radek.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-23-2025 06:10 AM
Hey Raked, thanks for replying.
Yes, indeed this is feasible solution but somehow the ADF Additional cluster settings adjusted at LK level are not bypass to the Job compute cluster 😞
