Databricks Community

minhhung0507

Dear Databricks experts,I encountered the following error in Databricks:`com.databricks.sql.transaction.tahoe.DeltaFileNotFoundException: [DELTA_EMPTY_DIRECTORY] No file found in the directory: gs://cimb-prod-lakehouse/bronze-layer/losdb/pl_message/_...

minhhung0507

Hi Databricks experts,We're experiencing unexpectedly high costs from Regional Standard Class A Operations in GCS while running a Databricks pipeline. The costs seem related to frequent metadata queries, possibly tied to Delta table operations.In las...

minhhung0507 · 12-18-2024

Hi Databricks Community,I'm encountering an issue with watermarks in Delta Live Tables that's causing data loss in my streaming pipeline. Let me explain my specific problem:Current SituationI've implemented watermarks for stateful processing in my De...

minhhung0507 · 12-05-2024

Hello Databricks Community,I am experiencing an issue with Delta Lake where the _delta_log files are not being deleted automatically in GCS bucket, even though I have set the table properties to enable this behavior. Here is the configuration I used:...

minhhung0507

Hi @saurabh18cs ,Thanks for that very detailed explanation. I will take note and continue to observe this case.

minhhung0507

Hi @saurabh18cs , Thank you for your explanation regarding the VACUUM operation and the error I encountered. I appreciate your insights.I would like to clarify further: why does the VACUUM feature sometimes delete files that are still necessary and b...

minhhung0507

Dear @VZLA ,Thank you so much for your detailed insights and recommendations!We truly appreciate your suggestion about visualizing and analyzing metadata operations—it’s an excellent idea that we can definitely apply to identify and prioritize optim...

minhhung0507

Hi @VZLA ,Thank you for your detailed suggestions and guidance.Here are some updates and points regarding our current setup:The primary workloads/jobs we use are streaming jobs running on Databricks.We’ve already implemented Auto Optimize and Auto C...

minhhung0507

Dear @VZLA , @Walter_C ,Thank you for providing us with several solutions regarding our high Google Cloud Storage costs. We have confirmed that the high costs are indeed coming from metadata operations triggered by Delta table activity, with the GET ...

Databricks Community

User Stats

User Activity

Issue with DeltaFileNotFoundException After Vacuum and Missing Data Changes in Delta Log

How to reduce cost of "Regional Standard Class A Operations"

Handling Dropped Records in Delta Live Tables with Watermark - Need Optimization Strategy

Delta Log Files in GCS Not Deleting Automatically Despite Configuration

Re: Issue with DeltaFileNotFoundException After Vacuum and Missing Data Changes in Delta Log

Re: Issue with DeltaFileNotFoundException After Vacuum and Missing Data Changes in Delta Log

Re: How to reduce cost of "Regional Standard Class A Operations"

Re: How to reduce cost of "Regional Standard Class A Operations"

Re: How to reduce cost of "Regional Standard Class A Operations"