topic [INSUFFICIENT_PERMISSIONS] Insufficient privileges: in Data Engineering

[INSUFFICIENT_PERMISSIONS] Insufficient privileges:

Tamizh035 — Wed, 23 Oct 2024 11:58:47 GMT

While reading csv file using spark and listing the files under a folder using data bricks utils, I am getting below error:

[INSUFFICIENT_PERMISSIONS] Insufficient privileges: User does not have permission SELECT on any file. SQLSTATE: 42501

File <command-1200317151894398>, line 2 1 spark_df = spark.read.csv("sample_csv.csv") ----> 2 display(spark_df)

File /databricks/python_shell/dbruntime/display.py:131, in Display.display(self, input, *args, **kwargs) 129 # This version is for Serverless + Spark Connect dogfooding. 130 elif self.spark_connect_enabled and isinstance(input, ConnectDataFrame): --> 131 self.display_connect_table(input, **kwargs) 132 elif isinstance(input, ConnectDataFrame):
------
But able to read and write csv files using pandas in the same environment.
Below is the code which I used:

import pandas as pd

df = pd.read_csv("sample_csv.csv")

sample_csv.csv is the file I created and I have "can manage" privilege on it.

Kindly help me with resolving the issue.

display(df)

spark_df = spark.read.csv("sample_csv.csv")

display(spark_df)

files = dbutils.fs.ls("path")

Re: [INSUFFICIENT_PERMISSIONS] Insufficient privileges:

Panda — Wed, 23 Oct 2024 12:07:55 GMT

@Tamizh035 , If your file in dbfs or external location or local folder ?
Use dbutils.fs.ls to verify that the path exists and you have access:

files = dbutils.fs.ls("dbfs:/path_to_your_file/") display(files)

Re: [INSUFFICIENT_PERMISSIONS] Insufficient privileges:

Tamizh035 — Thu, 24 Oct 2024 05:00:37 GMT

Hi @Panda

Listing the files using dbutils too is giving same error. I have created the folder and I have "can manager" privilege on it.

ExecutionError: An error occurred while calling o415.ls.
: org.apache.spark.SparkSecurityException: [INSUFFICIENT_PERMISSIONS] Insufficient privileges:
User does not have permission SELECT on any file. SQLSTATE: 42501

Re: [INSUFFICIENT_PERMISSIONS] Insufficient privileges:

mpalacio — Tue, 03 Dec 2024 14:48:09 GMT

I have the same issue, did you manage to solve it?

I have the Databricks extension well configured and my role has enough permissions. Everything used to work propertly but now when I run my notebooks is giving me this issue and 'no module named dbruntime' all the time.