Does column masking work with job clusters

[Harsha777]

Hi,

We are trying to experiment with the column masking feature.

Here is our use case:

1. We have added a masking function to one of the columns of a table
2. the table is part of a notebook with some transformation logic
3. the notebook is executed as part of a workflow with a job cluster
4. the job cluster configuration is updated to use dbr15.4LTS and also tried dbr16.0
5. the notebook fails while accessing the table with error 
   

We tried accessing the masked table with SQL warehouse, Serverless, All-purpose clusters (DBR15.4) and it worked fine without any issue.

So, the question is - are there any restrictions with the masking concept in using the job clusters? we couldnt find anything in the documentation.
Any inputs would be highly appreciated,

Regards,
Harsha

[Walter_C · Accepted solution]

It seems that the issue is that the cluster you are using in single user cluster, can you try to modify the job cluster and make it shared?

[Harsha777]

Thanks @Walter_C it worked on changing the configuration as you have suggested.
One follow up question -  what does "Shared" access mode mean for a job cluster, is there a way to share a job cluster with other workloads? if so, it would be really interesting and would be good to know and optimize our stack, could you please share any resources

[MoJaMa]

For this to work on single-user clusters you need the Filtering Fleet enabled which requires you to enable Serverless Compute (same infra as Serverless Workflows / Notebooks / DLT).

https://docs.databricks.com/en/release-notes/product/2024/october.html#fine-grained-access-control-o...

" In workspaces enabled for serverless compute, if a query is run on supported compute such as single user compute and the query accesses any of the following objects ..."

[Harsha777]

Serverless compute is already enabled on the workspace and  we have tested against serverless compute and it worked.

[Walter_C]

Hello, the shared cluster on a job will act the same as in an all purpose cluster, basically means that the cluster will be available for any user with permissions to it, in a job there will not be much actions to be done but when an action you are running requires Shared Cluster you have the opportunity to set it on the job.

Single user cluster means that only the user that is assigned to that cluster can run actions on it