Error Listing Delta Log on GCS in Databricks

[minhhung0507]

I am encountering an issue while working with a Delta table in Databricks. The error message is as follows:

java.io.IOException: Error listing gs://cimb-prod-lakehouse/bronze-layer/dbd/customer_info_update_request_processing/_delta_log/

This issue occurs when attempting to access the Delta table located at the specified GCS path. Could you please help me understand:

1. What might be causing this error?
2. How can I resolve it to successfully access the Delta table?

For context:

• The Delta table is stored on Google Cloud Storage (GCS).
• I have already configured the necessary credentials for GCS access in Databricks.

Any guidance or troubleshooting steps would be greatly appreciated.

Thank you!

[kamal_ch]

Ensure that the Databricks workspace has the necessary permissions to access the GCS bucket. Check if the service account used for Databricks has "Storage Object Viewer" or a similar role granted.

Verify that the path "gs://cimb-prod-lakehouse/bronze-layer/dbd/customer_info_update_request_processing/_delta_log/" exists and matches exactly with the resource on the bucket, including case sensitivity.

Double-check the GCS configuration (e.g., credentials, region settings) in the Databricks environment.

If none of these steps resolve the issue, a detailed inspection of logs or collaborating with the Databricks support team may be necessary for further investigation.