The error you're encountering, com.databricks.sql.transaction.tahoe.DeltaFileNotFoundException: [DELTA_EMPTY_DIRECTORY] No file found in the directory: gs://cimb-prod-lakehouse/bronze-layer/losdb/pl_message/_delta_log, indicates that the _delta_log directory is empty or missing, which is critical for Delta Lake operations. This issue can arise due to improper use of the VACUUM operation.

• The VACUUM operation in Delta Lake is used to remove old files that are no longer needed for the current state of the table. However, if the retention period is set too short, it can inadvertently delete files that are still needed for the Delta table's metadata and transaction log.
• The default retention period for VACUUM is 7 days. If you set a shorter retention period, you risk deleting files that are still required.
• If the _delta_log directory is missing or corrupted, Delta Lake cannot properly record transactions. This can lead to inconsistencies and errors during data ingestion and querying.

VACUUM my_table RETAIN 168 HOURS; -- Retain files for 7 days