Databricks Community

It seems that nothing is being loaded into the GOOGLE_APPLICATION_CREDENTIALS. From https://github.com/GoogleCloudDataproc/hadoop-connectors/blob/master/gcs/INSTALL.md   # The JSON keyfile of the service account used for GCS # access when google.clou...

Attached is the sample query generated by Power BI. Without the OR conditions the query runs within seconds.

I'm dealing with the same problem.Doesn't make any sense make a feature that should make our dlt jobs faster unusable in every context. Is there a explanation for this? In my concept, even if incremental process would be 1.1x faster it should option ...

@gchandra for example a table called "cars", if I remove the table from DLT pipeline and drop the table from catalog. Now if I change the schema of the table, and create the table again using the same table name "cars" through the same pipeline, Why ...

Hi, If you are looking for the reason of job failure you can navigate to view details tab-> logs to figure out the root cause of the failure.The blank screen with no events might be caused as you might have selected one of the DLT table.You can navig...

Hi, # Set the table name and clustering columns table_name = "feature_store_table" clustering_columns = ["column1", "column2"] # Build the SQL command sql_command = f"ALTER TABLE {table_name} CLUSTER BY ({', '.join(clustering_columns)})" # Execute ...

Hi, It seems you are setting the retention period to 0 hours. You need to first use this config spark.databricks.delta.retentionDurationCheck.enabled to false. and then set the retentention duration config as you used. You can do a DRY RUN to check i...

To move Delta tables from a Dev workspace to a Prod workspace, you can use a combination of Delta Lake features and Databricks APIs. Here's a high-level approach with some Python code to help you get started: Method 1: Using CLONE command For smaller...

Hi, Yes there was a maintainance release on 7th august which might have caused this issue. If you are still experience this issue, please file a support ticket.

Hi, Possible Causes and Solutions Network Configuration: The private connectivity setup might be affecting DLT clusters differently. Cluster Configuration: Ensure DLT clusters are properly sized for the workload.Consider using a larger driver node fo...

1. Deduplication in medallion architecture can be handled in bronze or silver layer.2. If keeping a complete history of all raw data, including duplicates, in the bronze layer, handle deduplication in the silver layer.3. If not keeping a complete his...