I have a use case where service principals will read .csv files from Azure Storage Account and create views from them. This used to work in our legacy environment but we are currently migrating to Unity Catalog and when we tested our existing jobs we ran into insufficient privileges error. Error message was that the "User does not have permission SELECT on any file." I read the documentation and found the following:
You must have SELECT privileges on the ANY FILE securable to read using the following patterns on Unity Catalog-enabled standard clusters:
- Cloud object storage using URIs.
https://learn.microsoft.com/en-gb/azure/databricks/data-governance/table-acls/any-file
This is exactly what we are doing. So I tested by manually granting the following:
GRANT SELECT ON ANY FILE to `first.last@domain.com`;
After this reading files succeeded. Now the issue is that we are managing our configuration with Terraform and I would like to do add this into the configuration as well. However, I couldn't find which resource I should use to create this privilege. Any idea how to achieve this using Terraform? Or is there a better way of doing this? This is the way teams in my organization have done things before, so I'm not sure if they are willing to change things that much.