topic Re: Error Listing Delta Log on GCS in Databricks in Data Engineering

Error Listing Delta Log on GCS in Databricks

minhhung0507 — Fri, 14 Feb 2025 03:11:56 GMT

I am encountering an issue while working with a Delta table in Databricks. The error message is as follows:

java.io.IOException: Error listing gs://cimb-prod-lakehouse/bronze-layer/dbd/customer_info_update_request_processing/_delta_log/

This issue occurs when attempting to access the Delta table located at the specified GCS path. Could you please help me understand:

What might be causing this error?
How can I resolve it to successfully access the Delta table?

For context:

The Delta table is stored on Google Cloud Storage (GCS).
I have already configured the necessary credentials for GCS access in Databricks.

Any guidance or troubleshooting steps would be greatly appreciated.

Thank you!

Re: Error Listing Delta Log on GCS in Databricks

kamal_ch — Thu, 15 May 2025 13:14:48 GMT

Ensure that the Databricks workspace has the necessary permissions to access the GCS bucket. Check if the service account used for Databricks has "Storage Object Viewer" or a similar role granted.

Verify that the path "gs://cimb-prod-lakehouse/bronze-layer/dbd/customer_info_update_request_processing/_delta_log/" exists and matches exactly with the resource on the bucket, including case sensitivity.

Double-check the GCS configuration (e.g., credentials, region settings) in the Databricks environment.

If none of these steps resolve the issue, a detailed inspection of logs or collaborating with the Databricks support team may be necessary for further investigation.