Hello @liu 

The fact that you're getting a permissions issue when trying to access the same location in different notebooks is indeed confusing. Here's my take on what could be the cause of this issue:

Jobs in databricks default to creating their own short-lived clusters. These are effectively temporary clusters, created specifically to enable a single job or task to run. If each task/ notebook within your job has its own ephemeral cluster, they also have some separate resources, i.e separate temp environments. I would then assume that when your second notebook tries to access the temp directory belonging to your first notebook, it runs into a permissions issue. Equally, it could be the case that the second notebook simply can't find the file path, and the issue simply manifests as a "permissions" issue.

The way to get around this would be to use a persistent, all-purpose compute resource, or write to non-local (global) file paths. For more information on compute resources, the following link may be useful:

https://docs.databricks.com/aws/en/jobs/compute

Please let me know if you find this helpful - feel free to ask any further questions. 

Regards - Pilsner