Hi @LearnDB123 ,
Yes, there have been significant changes to the handling of storage locations, including `/tmp/`, when using Unity Catalog in Databricks. Unity Catalog introduces stricter governance and security controls, which affect access to certain storage paths, including temporary directories.
Why You Are Facing Permission Denied Errors with `/tmp/` in Unity Catalog?
Unity Catalog enforces stricter access controls to prevent unauthorized access and ensure data governance. Access to certain locations, such as `/tmp/`, may be restricted by default because they are not managed or governed under Unity Catalogโs data security model. In Databricks with Unity Catalog, the `/tmp/` directory is no longer accessible in the same way as it was in non-UC environments. This is because Unity Catalog governs storage locations more strictly and `/tmp/` is considered outside the managed and secure storage zones defined by Unity Catalog.
Recommended Solutions
Use Managed Storage Locations in Unity Catalog. Instead of using `/tmp/`, you should configure and use Unity Catalog-managed storage locations for temporary file handling. This allows the data to be secured and audited correctly under Unity Catalogโs governance.
Set Up an External Location in Unity Catalog. Create an external location using Unity Catalog, which points to a specific path in your Azure Blob Storage. This location can then be used for temporary data operations and will respect Unity Catalogโs access permissions.
-- Create an external location in Unity Catalog
CREATE EXTERNAL LOCATION temp_storage
URL 'abfss://<your-container>@<your-storage-account>.dfs.core.windows.net/tmp/'
WITH (STORAGE CREDENTIAL my_storage_credential)
COMMENT 'Temporary storage location for data processing'
