Error Listing Delta Log on GCS in Databricks

minhhung0507
Valued Contributor

‎02-13-2025 07:11 PM

I am encountering an issue while working with a Delta table in Databricks. The error message is as follows:

java.io.IOException: Error listing gs://cimb-prod-lakehouse/bronze-layer/dbd/customer_info_update_request_processing/_delta_log/

minhhung0507_0-1739502695009.png

This issue occurs when attempting to access the Delta table located at the specified GCS path. Could you please help me understand:

  1. What might be causing this error?
  2. How can I resolve it to successfully access the Delta table?

For context:

  • The Delta table is stored on Google Cloud Storage (GCS).
  • I have already configured the necessary credentials for GCS access in Databricks.

Any guidance or troubleshooting steps would be greatly appreciated.

Thank you!

Regards,
Hung Nguyen
Labels:
0 Kudos

kamal_ch
Databricks Employee
Databricks Employee

‎05-15-2025 06:14 AM

Ensure that the Databricks workspace has the necessary permissions to access the GCS bucket. Check if the service account used for Databricks has "Storage Object Viewer" or a similar role granted.

Verify that the path "gs://cimb-prod-lakehouse/bronze-layer/dbd/customer_info_update_request_processing/_delta_log/" exists and matches exactly with the resource on the bucket, including case sensitivity.

Double-check the GCS configuration (e.g., credentials, region settings) in the Databricks environment.

If none of these steps resolve the issue, a detailed inspection of logs or collaborating with the Databricks support team may be necessary for further investigation.

0 Kudos