cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Does column masking work with job clusters

Harsha777
New Contributor III

Hi,

We are trying to experiment with the column masking feature.

Here is our use case:

  1. We have added a masking function to one of the columns of a table
  2. the table is part of a notebook with some transformation logic
  3. the notebook is executed as part of a workflow with a job cluster
  4. the job cluster configuration is updated to use dbr15.4LTS and also tried dbr16.0
  5. the notebook fails while accessing the table with error 
    Harsha777_0-1732696132629.pngHarsha777_1-1732696804007.png

We tried accessing the masked table with SQL warehouse, Serverless, All-purpose clusters (DBR15.4) and it worked fine without any issue.

So, the question is - are there any restrictions with the masking concept in using the job clusters? we couldnt find anything in the documentation.
Any inputs would be highly appreciated,

Regards,
Harsha

0 REPLIES 0

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group