topic Re: Serverless Cluster Issue in Data Engineering

Serverless Cluster Issue

Avinash_Narala — Wed, 17 Jul 2024 06:31:51 GMT

Hi,

While using Serverless cluster I'm not able to access dbfs files, saying I don't have permission to the file.

But while accessing them using All-purpose cluster I'm able to access them.

Why am I facing this issue?

Re: Serverless Cluster Issue

Vidhi_Khaitan — Tue, 13 May 2025 10:57:14 GMT

Hi Avinash,

This is an expected behaviour by design ->
https://docs.databricks.com/aws/en/compute/serverless/limitations#general-limitations

DBFS FUSE (i.e. file access through the local filesystem via /dbfs) is not available by design on serverless

Re: Serverless Cluster Issue

RobertWhite — Tue, 13 May 2025 12:05:25 GMT

You might be encountering this issue due to permission differences between the serverless and all-purpose clusters. Serverless environments often have restricted access for enhanced security. Make sure the appropriate IAM roles or access controls are granted. Also, if you're looking into detailed activity tracking or system event logs like Tennessee arrest database, similar permission hierarchies could affect your data visibility there as well. 4o