topic Does column masking work with job clusters in Data Engineering

Does column masking work with job clusters

Harsha777 — Wed, 27 Nov 2024 08:41:45 GMT

Hi,

We are trying to experiment with the column masking feature.

Here is our use case:

We have added a masking function to one of the columns of a table
the table is part of a notebook with some transformation logic
the notebook is executed as part of a workflow with a job cluster
the job cluster configuration is updated to use dbr15.4LTS and also tried dbr16.0
the notebook fails while accessing the table with error

We tried accessing the masked table with SQL warehouse, Serverless, All-purpose clusters (DBR15.4) and it worked fine without any issue.

So, the question is - are there any restrictions with the masking concept in using the job clusters? we couldnt find anything in the documentation.
Any inputs would be highly appreciated,

Regards,
Harsha

Re: Does column masking work with job clusters

Walter_C — Wed, 27 Nov 2024 14:35:05 GMT

It seems that the issue is that the cluster you are using in single user cluster, can you try to modify the job cluster and make it shared?

Re: Does column masking work with job clusters

MoJaMa — Wed, 27 Nov 2024 17:58:50 GMT

For this to work on single-user clusters you need the Filtering Fleet enabled which requires you to enable Serverless Compute (same infra as Serverless Workflows / Notebooks / DLT).

https://docs.databricks.com/en/release-notes/product/2024/october.html#fine-grained-access-control-on-single-user-compute-is-generally-available

" In workspaces enabled for serverless compute, if a query is run on supported compute such as single user compute and the query accesses any of the following objects ..."

Re: Does column masking work with job clusters

Harsha777 — Thu, 28 Nov 2024 09:23:33 GMT

Thanks @Walter_C it worked on changing the configuration as you have suggested.
One follow up question - what does "Shared" access mode mean for a job cluster, is there a way to share a job cluster with other workloads? if so, it would be really interesting and would be good to know and optimize our stack, could you please share any resources

Re: Does column masking work with job clusters

Harsha777 — Thu, 28 Nov 2024 09:25:58 GMT

Serverless compute is already enabled on the workspace and we have tested against serverless compute and it worked.

Re: Does column masking work with job clusters

Walter_C — Thu, 28 Nov 2024 17:17:45 GMT

Hello, the shared cluster on a job will act the same as in an all purpose cluster, basically means that the cluster will be available for any user with permissions to it, in a job there will not be much actions to be done but when an action you are running requires Shared Cluster you have the opportunity to set it on the job.

Single user cluster means that only the user that is assigned to that cluster can run actions on it