How do I create a job for a notebook not in the /Users/ directory?

[kpendergast]

I am setting up a job to to load data from S3 into Delta using Auto loader. I can do this fine in interactive mode. When trying to create a job in the UI. I can select the notebook in the root directory I created for the project within the create job UI however instead of it being "/abc/notebook" it uses "/Users/my_username_dir/notebook" and that doesn't exist. I have full admin privileges.

[Kaniz]

Hi @kpendergast! My name is Kaniz, and I'm the technical moderator here. Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question first. Or else I will get back to you soon. Thanks.

[kpendergast]

Thanks @Kaniz Fatma​  if there is a best practice for how to structure projects that would also be helpful. It just seems strange to need to have files that run streaming jobs and other parts of a pipeline live inside user directories.

[User16844513407 · Accepted solution]

Hi @Ken Pendergast​, you are supposed to be able to reference any Notebook you have the right permissions on so it looks like you are running into a bug, can you please reach out to support or email me directly with your workspace ID? My email is jan@databricks.com. Sorry for the inconvenience.

[kpendergast]

Thanks for the feedback. I had reported this to my SA Kristin and seems like it has been resolved. I just tested creating a new job on a file not in a user folder and it didnt show me an error when trying to save the task. I was not even able to save a task pointing to a file not in a user's directory.

Thanks again for responding I can now move all the job files out of my user directory 😀