- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-01-2021 12:34 PM
I am setting up a job to to load data from S3 into Delta using Auto loader. I can do this fine in interactive mode. When trying to create a job in the UI. I can select the notebook in the root directory I created for the project within the create job UI however instead of it being "/abc/notebook" it uses "/Users/my_username_dir/notebook" and that doesn't exist. I have full admin privileges.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-02-2021 03:05 AM
Thanks @Kaniz Fatma if there is a best practice for how to structure projects that would also be helpful. It just seems strange to need to have files that run streaming jobs and other parts of a pipeline live inside user directories.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-12-2021 04:23 AM
Hi @Ken Pendergast, you are supposed to be able to reference any Notebook you have the right permissions on so it looks like you are running into a bug, can you please reach out to support or email me directly with your workspace ID? My email is jan@databricks.com. Sorry for the inconvenience.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-12-2021 05:01 AM
Thanks for the feedback. I had reported this to my SA Kristin and seems like it has been resolved. I just tested creating a new job on a file not in a user folder and it didnt show me an error when trying to save the task. I was not even able to save a task pointing to a file not in a user's directory.
Thanks again for responding I can now move all the job files out of my user directory 😀
