Databricks

DanR · ‎12-13-2023

We are having intermittent errors where a Job Task cannot access a Catalog through a Volume, with the error: `PermissionError: [Errno 1] Operation not permitted: '/Volumes/mycatalog'`.The Job has 40 tasks running in parallel and every few runs we experience this error in a different Task. Our workspace is on Azure and is Terraformed.

Stack trace:

File /local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/watermark.py:137, in Watermark.write(self)
    135 if not os.path.exists(self.path_base):
    136     self.logger.debug(f"Creating directory: {self.path_base}")
--> 137     os.makedirs(self.path_base, exist_ok=True)
    139 while current_retry < self.__max_retries and not success:
    140     try:

File /usr/lib/python3.10/os.py:215, in makedirs(name, mode, exist_ok)
    213 if head and tail and not path.exists(head):
    214     try:
--> 215         makedirs(head, exist_ok=exist_ok)
    216     except FileExistsError:
    217         # Defeats race condition when another thread created the path
    218         pass

File /usr/lib/python3.10/os.py:215, in makedirs(name, mode, exist_ok)
    213 if head and tail and not path.exists(head):
    214     try:
--> 215         makedirs(head, exist_ok=exist_ok)
    216     except FileExistsError:
    217         # Defeats race condition when another thread created the path
    218         pass

    [... skipping similar frames: makedirs at line 215 (2 times)]

File /usr/lib/python3.10/os.py:215, in makedirs(name, mode, exist_ok)
    213 if head and tail and not path.exists(head):
    214     try:
--> 215         makedirs(head, exist_ok=exist_ok)
    216     except FileExistsError:
    217         # Defeats race condition when another thread created the path
    218         pass

File /usr/lib/python3.10/os.py:225, in makedirs(name, mode, exist_ok)
    223         return
    224 try:
--> 225     mkdir(name, mode)
    226 except OSError:
    227     # Cannot rely on checking for EEXIST, since the operating system
    228     # could give priority to other errors like EACCES or EROFS
    229     if not exist_ok or not path.isdir(name):

PermissionError: [Errno 1] Operation not permitted: '/Volumes/mycatalog'

Kaniz · ‎12-14-2023

Hi @DanR, The error message PermissionError: [Errno 1] Operation not permitted: '/Volumes/mycatalog' indicates that the Python process does not have the necessary permissions to create directories or perform certain operations on the specified path. This could be due to the operating system’s security settings or the user permissions of the process.

Here are a few potential solutions:

Check User Permissions: Ensure that the user running the Python process has the necessary permissions to access and modify the directory /Volumes/mycatalog.
Modify Security Settings: On some systems, you may need to modify the security settings to allow the.... For example, on macOS, you might need to give your terminal program access to the relevant folders. You can do this in System Preferences -> Privacy & Security -> Files and Folders.
Change Directory Location: If possible, try moving your files to a different directory where the Pyt....
Run as Root: As a last resort, you could try running the Python process as root, which should bypass any permission issues. However, this approach comes with its own security risks and is generally not recommended unless abs....

Also, since you mentioned that your workspace is on Azure and is Terraformed, you might want to chec....

I hope this helps! Let me know if you have any other questions.

Mohamed_Deyab · ‎03-21-2024

We are having a similar issue intermittently in Azure Databricks. I notice it happens when there is parallelism involved. The same wheel when run at non-heavy times, consistently finishes successfully. It only intermittently fail when the load is heavy.

Below is a sample error.

[Errno 13] Permission denied: '/local_disk0/spark-554a306a-e837-4180-a168-f5ee6e92bb75/trustedTemp-a74efb23-77ff-4d8f-b2bf-e96c708a0e96/tmpoz5zp7qn'"}