topic Re: Shared cluster in Administration & Architecture

Shared cluster

shiva1212 — Sat, 10 Aug 2024 22:10:48 GMT

Hello Community,

At my company, we use Databricks extensively for workflow development. We operate on a shared cluster that multiple users access concurrently, and we utilize a shared workspace, where one folder is accessible to a designated group. Each user manages their data through their individual secret scope.

However, I have observed a significant issue: when User A executes a code cell, User B is able to execute the following cell, effectively continuing from where User A left off. This situation compromises the isolation between user sessions, posing a potential security risk.

Can anyone suggest a way to ensure better isolation between user sessions in this shared environment?

Re: Shared cluster

Aviral-Bhardwaj — Sun, 11 Aug 2024 16:38:25 GMT

it should not happen , if the user is using different notebooks , if they are using same notebook, it is expected

Re: Shared cluster

shiva1212 — Sun, 18 Aug 2024 19:21:36 GMT

Thank you for your response, Aviral.

I agree that if users are using different notebooks, there shouldn’t be any security threat. However, the concern arises even when two users are using the same notebook; it should ideally be different sessions for each user.

Each user has their own OpenAI token. For instance, if User A runs a cell to extract their token, and then User B runs the next cell, User B ends up using User A’s token. This is undesirable behavior and poses a significant security risk.

From my understanding, a shared compute cluster should keep sessions separate for User A and User B. This means that while they share the code, they shouldn’t be sharing the variable values. Unfortunately, this separation is not happening in our environment.

Has anyone else experienced this, or does anyone have suggestions on how to enforce better session isolation?

Re: Shared cluster

Aviral-Bhardwaj — Mon, 19 Aug 2024 05:43:02 GMT

No Shiva, it don't work like that,one notebook one session simple, collaborative notebook means one development notebook with multiple user ,but multiple session wont work like that .