Databricks Community

 Hi Databricks Experts,I'm currently using Delta Live Table to generate master data managed within Unity Catalog, with the data stored directly in Google Cloud Storage. I then utilize Spark to read these master data from the GCS bucket. However, I’m ...

  We're encountering a performance issue with Spark SQL when working with data from Unity Catalog. Specifically, when I use Spark to read data from a Unity Catalog partition created by DLT and then create a view, the executor retrieval is very slow. ...

I'm encountering a puzzling schema merge issue with my Delta Live Table. My setup involves several master tables on Databricks, and due to a schema change in the source database, one of my Delta Live Tables has a column (e.g., "reference_score") that...

Hi everyone,I’m running multiple real-time pipelines on Databricks using a single job that submit them via thread pool. Most of the pipelines work fine, but a few of them occasionally get stuck for several hours, causing data loss. The challenge is t...

I am encountering an issue while working with a Delta table in Databricks. The error message is as follows:java.io.IOException: Error listing gs://cimb-prod-lakehouse/bronze-layer/dbd/customer_info_update_request_processing/_delta_log/ This issue occ...