Does column masking work with job clusters

Harsha777
New Contributor III

‎11-27-2024 12:41 AM

Hi,

We are trying to experiment with the column masking feature.

Here is our use case:

  1. We have added a masking function to one of the columns of a table
  2. the table is part of a notebook with some transformation logic
  3. the notebook is executed as part of a workflow with a job cluster
  4. the job cluster configuration is updated to use dbr15.4LTS and also tried dbr16.0
  5. the notebook fails while accessing the table with error 
    Harsha777_0-1732696132629.pngHarsha777_1-1732696804007.png

We tried accessing the masked table with SQL warehouse, Serverless, All-purpose clusters (DBR15.4) and it worked fine without any issue.

So, the question is - are there any restrictions with the masking concept in using the job clusters? we couldnt find anything in the documentation.
Any inputs would be highly appreciated,

Regards,
Harsha

0 Kudos

Walter_C
Databricks Employee
Databricks Employee

‎11-27-2024 06:35 AM

It seems that the issue is that the cluster you are using in single user cluster, can you try to modify the job cluster and make it shared?

 

View solution in original post

Harsha777
New Contributor III
In response to Walter_C

‎11-28-2024 01:23 AM

Thanks @Walter_C it worked on changing the configuration as you have suggested.
One follow up question -  what does "Shared" access mode mean for a job cluster, is there a way to share a job cluster with other workloads? if so, it would be really interesting and would be good to know and optimize our stack, could you please share any resources

0 Kudos

MoJaMa
Databricks Employee
Databricks Employee

‎11-27-2024 09:58 AM

For this to work on single-user clusters you need the Filtering Fleet enabled which requires you to enable Serverless Compute (same infra as Serverless Workflows / Notebooks / DLT).

https://docs.databricks.com/en/release-notes/product/2024/october.html#fine-grained-access-control-o...

" In workspaces enabled for serverless compute, if a query is run on supported compute such as single user compute and the query accesses any of the following objects ..."

0 Kudos

Harsha777
New Contributor III
In response to MoJaMa

‎11-28-2024 01:25 AM

Serverless compute is already enabled on the workspace and  we have tested against serverless compute and it worked.

0 Kudos

Walter_C
Databricks Employee
Databricks Employee

‎11-28-2024 09:17 AM

Hello, the shared cluster on a job will act the same as in an all purpose cluster, basically means that the cluster will be available for any user with permissions to it, in a job there will not be much actions to be done but when an action you are running requires Shared Cluster you have the opportunity to set it on the job.

Single user cluster means that only the user that is assigned to that cluster can run actions on it

 

0 Kudos