Databricks Community

Hi, Possible Causes and Solutions Network Configuration: The private connectivity setup might be affecting DLT clusters differently. Cluster Configuration: Ensure DLT clusters are properly sized for the workload.Consider using a larger driver node fo...

1. Deduplication in medallion architecture can be handled in bronze or silver layer.2. If keeping a complete history of all raw data, including duplicates, in the bronze layer, handle deduplication in the silver layer.3. If not keeping a complete his...

  The error in scenario 3 is likely due to the fact that the service principal is not an owner of the DLT pipeline that creates the materialized views. Even though the job is running on a shared cluster, the service principal still needs to be an own...

Hi, Can you please share what the error stack trace looks like? One possible cause of this error is that the schema of the table you are reading from does not match the schema of the data you are writing to.

Using DLT for Your Use Case DLT can be a good fit for your scenario, especially when implementing Slowly Changing Dimension (SCD) Type 2. Here's how you can approach this: Ingestion with Auto Loader: Use Auto Loader to ingest the daily parquet files ...

When you use `dlt.apply_changes` to update the silver table, it adds four hidden columns for tracking changes. These columns include `event_time`, `read_version`, `commit_version`, and `is_deleted`. When you run this process for the first time agains...

Hi, The best cluster type can depend on various factors, such as the specific requirements of your pipelines, the volume of data you're processing, and your budget. Therefore, it's always a good idea to test different cluster types and configurations...

@Retired_mod Please help me with this situation. I really need your help here. Through out the exam my eyes were on the screen only. 

To add a CreatedDate column that captures the timestamp when a record is first inserted into the table, you can modify your Delta Live Tables (DLT) pipeline setup as follows: 1) Define the schema for your streaming table to include the CreatedDate co...

Hey @DBUser2 It looks that delete is not supported at all based on documentation: Write-backThe Simba Apache Spark ODBC Connector supports translation for the followingsyntax when connecting to a Spark Thrift Server instance that is running Spark 1.3...

Integrating a Databricks notebook with a Git repository using a feature branch sounds like a clean workflow for version control. The step-by-step process makes it straightforward to collaborate and track changes effectively. It's great that Databrick...

In DBFS, the checkpoint directory is automatically created when you set it using spark.sparkContext.setCheckpointDir(checkpoint_dir). This means that you do not need to explicitly create the directory beforehand using dbutils.fs.mkdirs(checkpoint_dir...

There are no specific column datatype constraints for the first 32 columns used for the statistics that help with data skipping in Databricks. However, data skipping is not supported for partition columns.