Saving a file to /tmp is not working after migration to Unity Catalog

LearnDB123 — Wed, 04 Sep 2024 16:14:43 GMT

Hi,

We upgraded our runtime cluster to Unity Catalog recently and since some of the code has been failing which was working fine earlier.

We used to save files to "/tmp/" and then move them from temp into our blob storage however since the migration to UC this has been failing with an error of Permission Denied.

Has there been any changes to the tmp storage location with UC?

Re: Saving a file to /tmp is not working after migration to Unity Catalog

filipniziol — Thu, 05 Sep 2024 18:05:57 GMT

Hi @LearnDB123 ,

Yes, there have been significant changes to the handling of storage locations, including `/tmp/`, when using Unity Catalog in Databricks. Unity Catalog introduces stricter governance and security controls, which affect access to certain storage paths, including temporary directories.

Why You Are Facing Permission Denied Errors with `/tmp/` in Unity Catalog?

Unity Catalog enforces stricter access controls to prevent unauthorized access and ensure data governance. Access to certain locations, such as `/tmp/`, may be restricted by default because they are not managed or governed under Unity Catalog’s data security model. In Databricks with Unity Catalog, the `/tmp/` directory is no longer accessible in the same way as it was in non-UC environments. This is because Unity Catalog governs storage locations more strictly and `/tmp/` is considered outside the managed and secure storage zones defined by Unity Catalog.

Recommended Solutions

Use Managed Storage Locations in Unity Catalog. Instead of using `/tmp/`, you should configure and use Unity Catalog-managed storage locations for temporary file handling. This allows the data to be secured and audited correctly under Unity Catalog’s governance.

Set Up an External Location in Unity Catalog. Create an external location using Unity Catalog, which points to a specific path in your Azure Blob Storage. This location can then be used for temporary data operations and will respect Unity Catalog’s access permissions.

-- Create an external location in Unity Catalog CREATE EXTERNAL LOCATION temp_storage URL 'abfss://<your-container>@<your-storage-account>.dfs.core.windows.net/tmp/' WITH (STORAGE CREDENTIAL my_storage_credential) COMMENT 'Temporary storage location for data processing'

Re: Saving a file to /tmp is not working after migration to Unity Catalog

Rahul6 — Fri, 16 May 2025 02:25:10 GMT

Hi @filipniziol

Could we use volumes for this temp processing rather than doing S3

topic Re: Saving a file to /tmp is not working after migration to Unity Catalog in Data Engineering

Saving a file to /tmp is not working after migration to Unity Catalog

Re: Saving a file to /tmp is not working after migration to Unity Catalog

Re: Saving a file to /tmp is not working after migration to Unity Catalog