- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-07-2022 07:42 AM
Hey Kaniz, Thanks for your answer.
I'm not sure you understood me - I will try to make it more clear.
I'm trying to automate ML training process for our developers using databricks.
When developer finish with all his code - I packaging it into a wheel file and upload it into databricks file system, This package have an entry points let's call it train for now.
I managed to create a job using the CLI with all the needed configuration but when im trying to run the job im getting an error - cluster Manage access needed to install cluster libraries.
All the code in the wheel file are .py files.
In the job_config.json file i declared on libraries needed for the job to run ( e.g. the wheel file that have been already uploaded)
Is there a way to run the job without getting Manage access error? - to install the library just for the job scope and not for the cluster? ( like notebook scope lib)
Hope its more clear now, If not let me know and i will try to explain better