topic Re: Can't query delta tables, token missing required scope in Data Engineering

Can't query delta tables, token missing required scope

Miro_ta — Sat, 04 Nov 2023 18:03:53 GMT

Hello,
I've correctly set up a stream from kinesis, but I can't read anything from my delta table

I'm actually reproducing the demo from Frank Munz: https://github.com/fmunz/delta-live-tables-notebooks/tree/main/motion-demo

and I'm running the following script which is named M-Magn SSS.py

# Databricks notebook source # MAGIC %md # MAGIC ![Name of the image](https://raw.githubusercontent.com/fmunz/motion/master/img/magn.png) # COMMAND ---------- # MAGIC %sql # MAGIC USE demo_frank.motion; # MAGIC DESCRIBE table sensor; # COMMAND ---------- from pyspark.sql.functions import window # set watermark for 1 minute for time column # spark.sql.shuffle.partitions (default 200, match number of cores) spark.conf.set("spark.sql.shuffle.partitions", 30) display(spark.readStream.format("delta"). table("sensor").withWatermark("time", "3 seconds"). \ groupBy(window("time", "1 second")).avg("magn").orderBy("window",ascending=False).limit(30))

I've created the pipeline successfully, I'm now trying to look at my delta table, but databricks raises the following error when i run this:

%sql USE bigdatalab.motion; DESCRIBE table sensor;

Error in SQL statement: ExecutionException: org.apache.spark.sql.AnalysisException: 403: Your token is missing the required scopes for this endpoint.

I don't know what could be the cause of the problem, I've tried to set up all the authorizations but it doesn't work.

Hope you can help me, I'm using the Databricks on AWS 14 days trial.

Re: Can't query delta tables, token missing required scope

Miro_ta — Sat, 04 Nov 2023 18:01:15 GMT

If I try to query any other table that is not delta, I don't get any error.

Re: Can't query delta tables, token missing required scope

trejas — Thu, 09 Nov 2023 01:47:57 GMT

Did you figure out an answer to this? I am seeing the same thing on a delta streaming table.

Re: Can't query delta tables, token missing required scope

Miro_ta — Fri, 10 Nov 2023 19:27:35 GMT

Not really, but I solved in my case using the hive metastore instead of the unity catalog, maybe it's because I'm on a trial version

Re: Can't query delta tables, token missing required scope

nag_kanchan — Thu, 04 Jan 2024 18:38:37 GMT

I am facing the same issue but i cannot move my data to hive_metastore. Does anyone has a solution for Unity catalog?

@Retired_mod

Re: Can't query delta tables, token missing required scope

jlb0001 — Sat, 20 Jan 2024 18:55:06 GMT

I'm running into a similar situation such that DESCRIBE <table> on a table in UC works fine but running DESCRIBE EXTENDED <table> on that same table gives me that same error from a databricks notebook.

Strangely, I don't get this error if I switch the notebook to instead use a Serverless SQL Warehouse, and not a personal cluster compute.

So, I personally definitely have whatever access is needed, but somehow this compute instance does not have access to just the extended properties?

ExecutionException: org.apache.spark.sql.AnalysisException: 403: Your token is missing the required scopes for this endpoint.

What permissions must the compute instance have to access extended properties?

Re: Can't query delta tables, token missing required scope

udi_azulay — Sun, 21 Jan 2024 05:35:39 GMT

I am having similar issue, any update on this one ?

Re: Can't query delta tables, token missing required scope

Brosenberg — Wed, 21 Feb 2024 13:13:32 GMT

At this time, Delta Live Tables in Unity Catalog can only be read using a Shared compute or SQL Warehouse (support to read from Assigned compute is on the roadmap).

To read the table using Assigned compute (e.g. Personal Compute), you will first need to make a copy of the table using a Shared Compute/SQL Warehouse (i.e. not in a DLT pipeline). You can use Assigned compute to read from the copy.

Re: Can't query delta tables, token missing required scope

holly — Fri, 08 Mar 2024 10:36:52 GMT

Hello, I also had this issue. It was because I was trying to read a DLT table with a Machine Learning Runtime. At time of writing, Machine Learning Runtimes are not compatible with shared access mode, so I ended up setting up two clusters, one MLR assigned access mode, then a normal runtime shared access mode, and then played a fun game of dependency jenga.

I was only setting up a demo, so no big deal. If you're looking at something in production, I'm not sure I could recommend this approach unless it's a last resort.