Databricks Community

surajitDE

Hi Folks,I wanted to check if there’s a way to assign an email notification group to a Delta Live Tables (DLT) job for notifications.I know that it’s possible to configure Teams workflows and email notification groups for Databricks jobs, but in the ...

surajitDE · 06-18-2025

Hi folks,During the recent Data + AI Summit, there was a mention of a new real-time streaming mode in Delta Live Tables (DLT) that enables sub-300 millisecond latency. This sounds really promising!Could someone please guide me on:How do we enable thi...

surajitDE · 05-28-2025

Hi folks,How to change streaming table/column description in DLT during run time like we do for delta tables because ALTER STREAMING table isn't working.eg:COMMENT ON COLUMN ops_catalog_gld_dev.schema_silver.table_name.property_sid IS 'The key of the...

surajitDE · 03-28-2025

My DLT jobs are experiencing throttling due to the following error message:[GC (GCLocker Initiated GC) [PSYoungGen: 5431990K->102912K(5643264K)] 9035507K->3742053K(17431552K), 0.1463381 secs] [Times: user=0.29 sys=0.00, real=0.14 secs]I came across s...

surajitDE · 03-16-2025

@dlt.tabledef joined_table(): dim_df = spark.read.table("dim_table") # Reloads every batch fact_df = spark.readStream.table("fact_stream") return fact_df.join(dim_df, "id", "left")

surajitDE · 09-14-2025

if you add these settings in the pipeline JSON, the issue should get fixed:"pipelines.setMigrationHints" = "true""pipelines.enableDPMForExistingPipeline" = "true"I tried it on my side, and now it no longer throws the materialization error.

surajitDE · 03-21-2025

@Dlt.table(name=table_name,comment="just_testing",table_properties={"quality": "gold","mergeSchema": "true"},cluster_by=["test_id", "find_date"] # Optimizes for queries filtering on these columns)def testing_table():return create_testing_table(df_fin...

surajitDE · 03-21-2025

what is the Python syntax for adding liquid clustering to a Delta Live Table???

surajitDE · 03-21-2025

In a Delta Live Tables (DLT) continuous pipeline, does it make a difference if df_dim_prev (loaded in cell 1) is only read once at the start?For example, if df_dim_prev is initialized as:# Cell 1: Read dim_table oncedf_dim_prev = spark.read.table("di...

surajitDE · 03-16-2025

the question is default DLT pipeline refresh time is 5seconds but if I use combination of streaming and non streaming data then will it still be 5 seconds?

Databricks Community

User Stats

User Activity

Question on assigning email_notification_group to DLT Job Notifications?

How to Enable Sub-300 Millisecond Real-Time Mode in Delta Live Tables (DLT)

How to change streaming table/column description in DLT

How can we change from GC to G1GC in serverless

DLT refresh time for combination of streaming and non streaming tables?

Re: Delta Live Tables: dynamic schema

Re: Is it possible to use Liquid Clustering on Delta Live Tables / Materialised Views?

Re: Is it possible to use Liquid Clustering on Delta Live Tables / Materialised Views?

Re: DLT refresh time for combination of streaming and non streaming tables?

Re: DLT refresh time for combination of streaming and non streaming tables?