- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-04-2023 10:58 AM - edited 11-04-2023 11:03 AM
Hello,
I've correctly set up a stream from kinesis, but I can't read anything from my delta table
I'm actually reproducing the demo from Frank Munz: https://github.com/fmunz/delta-live-tables-notebooks/tree/main/motion-demo
and I'm running the following script which is named M-Magn SSS.py
# Databricks notebook source
# MAGIC %md
# MAGIC 
# COMMAND ----------
# MAGIC %sql
# MAGIC USE demo_frank.motion;
# MAGIC DESCRIBE table sensor;
# COMMAND ----------
from pyspark.sql.functions import window
# set watermark for 1 minute for time column
# spark.sql.shuffle.partitions (default 200, match number of cores)
spark.conf.set("spark.sql.shuffle.partitions", 30)
display(spark.readStream.format("delta").
table("sensor").withWatermark("time", "3 seconds"). \
groupBy(window("time", "1 second")).avg("magn").orderBy("window",ascending=False).limit(30))
I've created the pipeline successfully, I'm now trying to look at my delta table, but databricks raises the following error when i run this:
%sql
USE bigdatalab.motion;
DESCRIBE table sensor;
Error in SQL statement: ExecutionException: org.apache.spark.sql.AnalysisException: 403: Your token is missing the required scopes for this endpoint.
I don't know what could be the cause of the problem, I've tried to set up all the authorizations but it doesn't work.
Hope you can help me, I'm using the Databricks on AWS 14 days trial.