Error with file events in external location (via pub/sub)

hmnilp — Fri, 18 Jul 2025 09:59:46 GMT

Hi everyone!

I am new here and in Databricks world.

I was trying to enable a trigger event on file uploading in GCP bucket in order to start a job following this tutorial: https://docs.databricks.com/gcp/en/connect/unity-catalog/cloud-storage/manage-external-locations#-recommended-enable-file-events-for-an-external-location

So I had created a role with those permissions and then I assigned that role to the service account of the storage credentials. An important note: I didn't find that SA in the IAM entities list so I had to use the "grant access" button, insert the interested service account and assign the role created to it.

After all the operations listed, in the WS dashboard I had enabled the file events for the external location and I tested the connection with this result:

In the GCP log explorer I saw these logs:

Where we can notice that the topic was created, the subscription was created too and then a non-databricks service account tries to access and brokes everything. The logs after are a repetition of: service account that tries to access, databricks that tries to create a topic and a subscription that already exists.

In IAM entities list I found the only service account possible for that pattern:

Finally the log of the error on the service account is the following json (anonymized with *):

{ "protoPayload": { "@type": "type.googleapis.com/google.cloud.audit.AuditLog", "status": { "code": 7, "message": "The service account 's...3@gs...' does not have permission to publish messages to to the Cloud Pub/Sub topic '//pubsub.googleapis.com/projects/*/topics/csms-topic-by-path-*', or that topic does not exist." }, "authenticationInfo": { "principalEmail": "db-uc-credential-*-*@uc-*.iam.gserviceaccount.com" }, "requestMetadata": { "callerIp": "gce-internal-ip", "callerSuppliedUserAgent": "gcloud-java/unresolved Google-API-Java-Client/1.34.0 Google-HTTP-Java-Client/1.43.3 (gzip),gzip(gfe)", "requestAttributes": { "time": "2025-07-18T08:06:53.934327849Z", "auth": {} }, "destinationAttributes": {} }, "serviceName": "storage.googleapis.com", "methodName": "storage.buckets.update", "authorizationInfo": [ { "resource": "projects/_/buckets/*", "permission": "storage.buckets.update", "granted": true, "resourceAttributes": {} } ], "resourceName": "projects/_/buckets/*", "resourceLocation": { "currentLocations": [ "*" ] } }, "insertId": "17ucpftequabn", "resource": { "type": "gcs_bucket", "labels": { "bucket_name": "*", "project_id": "*", "location": "*" } }, "timestamp": "2025-07-18T08:06:53.928213630Z", "severity": "ERROR", "logName": "projects/*/logs/cloudaudit.googleapis.com%2Factivity", "receiveTimestamp": "2025-07-18T08:06:55.756659669Z" }

(the topic obviously exists)

Is it correct that uses that service account instead of the one created by databricks? Am I missing something?

Thanks in advance for any help!

Re: Error with file events in external location (via pub/sub)

Khaja_Zaffer — Tue, 22 Jul 2025 05:45:13 GMT

Hey @hmnilp

did you contact databricks, may be there is a feature to be enabled.

Re: Error with file events in external location (via pub/sub)

hmnilp — Wed, 23 Jul 2025 14:10:07 GMT

Hi! Thanks in advance for your answer.
No I didn't contact databricks. I thought that I was missing a piece somewhere.

I will do it for sure!

topic Re: Error with file events in external location (via pub/sub) in Administration & Architecture

Error with file events in external location (via pub/sub)

Re: Error with file events in external location (via pub/sub)

Re: Error with file events in external location (via pub/sub)