Databricks Community

aonurdemir

EnvironmentCloud: AWSCompute: ServerlessTable: a_big_tableTable type: Streaming Table (SDP pipeline)Table size: 641 GB, 6,210 filesLiquid Clustering columns: [event_time, integer_userId]delta.dataSkippingStatsColumns:event_time, integer_userId, integ...

aonurdemir · 03-05-2026

(Dear Moderators, why do you remove this question? It is a genuine question. Do not, please. )We have a continuous dlt pipeline that has tables updating every minute and partitioned by "partition_key" column. Table is 4 TB. Has 16k files. Sometimes w...

aonurdemir · 10-27-2025

Suddenly at "2025-10-23T14:12:48.409+00:00", coming file paths from file notification queue started to be urlencoded. Hence, our pipeline gets file not found exception. I think something has changed suddenly and broke notification system. Here are th...

aonurdemir · 09-15-2025

Hello everyone,I’m working with two CDC tables:table_x: 23,467,761 rows (and growing)table_y: 27,868,173,722 rowsMy goal is to build an SCD Type 2 table (table_z) using the Auto CDC API.The workflow I’d like to achieve is:Initial Load: Populate table...

aonurdemir · 02-23-2025

What is this legacy pipeline warning? I cannot find anything related on the internet and related forums. Thanks in advance.

aonurdemir · 11-04-2025

Hello @K_Anudeep ,As I mentioned, we realized that the special character encoding and decoding was broken. So we had solved the issue as the following by changing the path string in autoloader:old path: s3://my-sink/prod/app_daily/*/*/*/*.json.gznew ...

aonurdemir · 09-16-2025

I have solved it with the name parameter as this:dlt.create_streaming_table(name="table_z")dlt.create_auto_cdc_flow(name="backfill",target="table_z",source="table_y",keys=["user_id"],sequence_by=col("source_ts_ms"),ignore_null_updates=False,apply_as_...

aonurdemir · 02-25-2025

Ah I've found the issue. There is one more diff:This is the new key-value for target schema: "schema": "dev", This is the old key-value for target schema: "target": "prod" After changing the key from "target" to "schema", the warning has gone....

aonurdemir · 02-25-2025

I realized that creating pipeline button has changed. Databricks has brought new options to the create pipeline button. After selecting create etl option, the new json configuration is seen as the following:{ "development": true, "continuous"...

aonurdemir · 02-24-2025

Hi @Alberto_Umana,I provision it via UI. And, this is the json config of it:{ "id": "...", "pipeline_type": "WORKSPACE", "development": true, "continuous": false, "channel": "PREVIEW", "libraries": [ ... ], "nam...

Databricks Community

User Stats

User Activity

Liquid Clustering file pruning breaks when filtering on a high NULL numeric column in dataSkipping

Conflict between Predictive Optimization and High Frequency Writes

Broken s3 file paths in File Notifications for auto loader

Creating an SCD Type 2 Table with Auto CDC API (One-Time Load + Ongoing Updates)

Legacy Pipeline?

Re: Broken s3 file paths in File Notifications for auto loader

Re: Creating an SCD Type 2 Table with Auto CDC API (One-Time Load + Ongoing Updates)

Re: Legacy Pipeline?

Re: Legacy Pipeline?

Re: Legacy Pipeline?