I am running a Databricks job with a runas principal using Serverless compute. The job has a single notebook task. The notebook runs successfully on a job compute or when I run (as me) in the Workspace (still using serverless compute). However, I get a strange error when running as a service principal via the job:
SparkConnectGrpcException: 403 Forbidden: 403: Invalid access token. [ReqId: a4202901-13bb-40c0-8bdf-718db9f5da63] [Trace ID: 00-6433c827e5ef437784f44e8558df32ef-ae4061a78da73c36-00]
File <command-6464347790329852>, line 13 11 if debug_mode: 12 print('Dummy spark sql...') ---> 13 spark.sql(f"SELECT 1 as myValue"); 16 if debug_mode: 17 print('About to declare var_destination_database_name...')
This notebook used to result in a different strange error:
I was able to workaround that error by adding the principal to the "admins" group. Not ideal, but it worked. While attempting to determine the fine-grained permissions required, I accidentally deleted the Principal from the workspace. After re-adding the principal (and granting permissions to catalog/schema), I started getting the new error noted above.
This exact same code and similar permissions still work in other environments. If the principal is in admins group, it works. If not in admins, I get the "SELECT on any file" error. Only this environment has the token error. This seems like it is potentially a bug in how serverless runs??
The code that causes the error is the first "spark.sql" statement. In this case, I added a dummy line:
spark.sql("SELECT 1 as myValue")