Databricks Community

hmnilp · ‎07-18-2025

Hi everyone!

I am new here and in Databricks world.

I was trying to enable a trigger event on file uploading in GCP bucket in order to start a job following this tutorial: https://docs.databricks.com/gcp/en/connect/unity-catalog/cloud-storage/manage-external-locations#-re...

So I had created a role with those permissions and then I assigned that role to the service account of the storage credentials. An important note: I didn't find that SA in the IAM entities list so I had to use the "grant access" button, insert the interested service account and assign the role created to it.

After all the operations listed, in the WS dashboard I had enabled the file events for the external location and I tested the connection with this result:

In the GCP log explorer I saw these logs:

Where we can notice that the topic was created, the subscription was created too and then a non-databricks service account tries to access and brokes everything. The logs after are a repetition of: service account that tries to access, databricks that tries to create a topic and a subscription that already exists.

In IAM entities list I found the only service account possible for that pattern:

Finally the log of the error on the service account is the following json (anonymized with *):

{
  "protoPayload": {
    "@type": "type.googleapis.com/google.cloud.audit.AuditLog",
    "status": {
      "code": 7,
      "message": "The service account 's...3@gs...' does not have permission to publish messages to to the Cloud Pub/Sub topic '//pubsub.googleapis.com/projects/*/topics/csms-topic-by-path-*', or that topic does not exist."
    },
    "authenticationInfo": {
      "principalEmail": "db-uc-credential-*-*@uc-*.iam.gserviceaccount.com"
    },
    "requestMetadata": {
      "callerIp": "gce-internal-ip",
      "callerSuppliedUserAgent": "gcloud-java/unresolved Google-API-Java-Client/1.34.0 Google-HTTP-Java-Client/1.43.3 (gzip),gzip(gfe)",
      "requestAttributes": {
        "time": "2025-07-18T08:06:53.934327849Z",
        "auth": {}
      },
      "destinationAttributes": {}
    },
    "serviceName": "storage.googleapis.com",
    "methodName": "storage.buckets.update",
    "authorizationInfo": [
      {
        "resource": "projects/_/buckets/*",
        "permission": "storage.buckets.update",
        "granted": true,
        "resourceAttributes": {}
      }
    ],
    "resourceName": "projects/_/buckets/*",
    "resourceLocation": {
      "currentLocations": [
        "*"
      ]
    }
  },
  "insertId": "17ucpftequabn",
  "resource": {
    "type": "gcs_bucket",
    "labels": {
      "bucket_name": "*",
      "project_id": "*",
      "location": "*"
    }
  },
  "timestamp": "2025-07-18T08:06:53.928213630Z",
  "severity": "ERROR",
  "logName": "projects/*/logs/cloudaudit.googleapis.com%2Factivity",
  "receiveTimestamp": "2025-07-18T08:06:55.756659669Z"
}

(the topic obviously exists)

Is it correct that uses that service account instead of the one created by databricks? Am I missing something?

Thanks in advance for any help!