topic Re: How do I create a job for a notebook not in the /Users/ directory? in Data Engineering

How do I create a job for a notebook not in the /Users/ directory?

kpendergast — Mon, 01 Nov 2021 19:34:31 GMT

I am setting up a job to to load data from S3 into Delta using Auto loader. I can do this fine in interactive mode. When trying to create a job in the UI. I can select the notebook in the root directory I created for the project within the create job UI however instead of it being "/abc/notebook" it uses "/Users/my_username_dir/notebook" and that doesn't exist. I have full admin privileges.

Re: How do I create a job for a notebook not in the /Users/ directory?

kpendergast — Tue, 02 Nov 2021 10:05:25 GMT

Thanks @Kaniz Fatma if there is a best practice for how to structure projects that would also be helpful. It just seems strange to need to have files that run streaming jobs and other parts of a pipeline live inside user directories.

Re: How do I create a job for a notebook not in the /Users/ directory?

User16844513407 — Fri, 12 Nov 2021 12:23:03 GMT

Hi @Ken Pendergast, you are supposed to be able to reference any Notebook you have the right permissions on so it looks like you are running into a bug, can you please reach out to support or email me directly with your workspace ID? My email is jan@databricks.com. Sorry for the inconvenience.

Re: How do I create a job for a notebook not in the /Users/ directory?

kpendergast — Fri, 12 Nov 2021 13:01:23 GMT

Thanks for the feedback. I had reported this to my SA Kristin and seems like it has been resolved. I just tested creating a new job on a file not in a user folder and it didnt show me an error when trying to save the task. I was not even able to save a task pointing to a file not in a user's directory.

Thanks again for responding I can now move all the job files out of my user directory 😀