topic Re: How do you enable verbose logging from with in Workspace Settings using Terraform? in Data Engineering

How do you enable verbose logging from with in Workspace Settings using Terraform?

njglen — Tue, 21 Mar 2023 13:19:26 GMT

I've searched in the databricks provider and online and couldn't find out if it is possible to set the `Verbose Audit Logs` to `enabled` using Terraform. Can anybody clarify if it is possible?

Re: How do you enable verbose logging from with in Workspace Settings using Terraform?

Anonymous — Thu, 23 Mar 2023 04:09:08 GMT

@Nicholas Glen :

Yes, it is possible to enable verbose audit logs in Databricks Workspace Settings using Terraform.

You can use the databricks_workspace_cluster_policy resource to create or update a cluster policy that enables verbose audit logs. Here's an example code snippet:

resource "databricks_workspace_cluster_policy" "example_policy" {
  policy_id   = "example_policy"
  policy_name = "Example Policy"
  policy_json = jsonencode({
    "audit_logs": {
      "log_all_users": true,
      "log_all_clusters": true,
      "audit_enabled": true,
      "audit_logs_level": "ALL"
    }
  })
}

In this example, we're creating a new cluster policy with the ID example_policy that enables verbose audit logs for all users and clusters. The policy_json field specifies the JSON representation of the policy, which is a nested object with an audit_logs field that contains the settings for audit logging.

You can then attach this policy to a workspace by using the databricks_workspace_resource

resource with the policy field set to the ID of the policy:

resource "databricks_workspace_resource" "example_workspace" {
  name  = "example_workspace"
  path  = "/"
  policy {
    policy_id = databricks_workspace_cluster_policy.example_policy.policy_id
  }
}

This attaches the example_policy policy to the workspace with the name example_workspace.

Note that enabling verbose audit logs can generate a large amount of log data, so make sure you have adequate storage and log management in place.

Re: How do you enable verbose logging from with in Workspace Settings using Terraform?

Anonymous — Mon, 27 Mar 2023 04:58:48 GMT

Hi @Nicholas Glen

Thank you for posting your question in our community!

To assist you better, please take a moment to review the answer and let me know if it best fits your needs.

Please help us select the best solution by clicking on "Select As Best" if it does.

Your feedback will help us ensure that we are providing the best possible service to you. Thank you!

Re: How do you enable verbose logging from with in Workspace Settings using Terraform?

njglen — Mon, 27 Mar 2023 11:06:32 GMT

I've searched and there is no such resource as `databricks_workspace_cluster_policy`

Rather this should work:

resource "databricks_cluster_policy" "audit_logs_policy" {
  name = "Audit Logs Policy"
  definition = jsonencode({
    "audit_logs": {
      "audit_enabled": true,
      "audit_logs_level": "ALL"
      "log_all_clusters": true,
      "log_all_users": true,      
    }
  })
}
 
output "audit_policy_id" {
  value = databricks_cluster_policy.audit_logs_policy.id
}

This cluster policy will need to be applied to the cluster created inside the workspace.

Re: How do you enable verbose logging from with in Workspace Settings using Terraform?

qiaochu — Fri, 22 Sep 2023 02:26:23 GMT

The switch you're looking for is enableVerboseAuditLogs in databricks_workspace_conf

resource: {
  databricks_workspace_conf: {
    this: {
      custom_config: {
        enableIpAccessLists: true,
        enableVerboseAuditLogs: true,
      },
    },
  },