Hey guys,

I'm trying to create a job via databricks cli, This job is going to use a wheell file that I already upload to dbfs and exported from this package the entry point that needed for the job.

In the UI I can see that the job has been created, But when I'm trying to run this job Im getting an error that i need Manage access in order to install libraries on a cluster( Cluster scooped libraries) .

My questions are -

1. Is there a way to create a job via databricks cli and to install packages but in a Notebook scooped manner? ( Without getting a Manage access for a cluster)
2. Let's say instead of using existing cluster, while creating the job i will create a new cluster - should there be any problems to install libraries with this way?

my job_config.json file:

{
  "name": "test_databricks_cli_jobs",
  "tasks": [
    {"task_key": "Test_train_entrypoint",
      "description": "test print in train entrypoint",
      "depends_on": [],
      "existing_cluster_id": "Myicluster-id",
      "python_wheel_task": {
        "package_name":  "testpack",
        "entry_point": "train",
        "parameters": ["Random", "This is a test message"]
      },
      "libraries": [
      {"whl": "/dbfs/FileStore/jars/test/testpack-0.0.1-py3-none-any.whl"}
  ]
 
    }
  ]
 
}

command to deploy the job:

databricks jobs create --json-file job_config.json --version=2.1

Hope some one can help me.

Thanks!