- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-11-2024 10:48 PM
Hi @Ian_P , Thanks for bringing up your concerns, always happy to help 😁
Are you getting the py4j.security exeception when trying to schedule a job on the shared cluster?
If yes then could you please share the screenshot of the notebook code that you are scheduling as well? I would like to understand what the code in the Notebook is doing that is encountering the Py4JSecurity. Due to the Table access control and Py4j security, there are some limitations on Shared mode clusters like commands that access the SparkContext.
Please refer the below doc for the more details about the limitations:
https://docs.databricks.com/en/data-governance/unity-catalog/compute.html#shared-access-mode-limitat...
Looking forward to hearing from you!