Databricks Community

Dhruv-22

The following is a minimal reproducible example of what I'm facing right now.%sql CREATE OR REPLACE TABLE edw_nprd_aen.bronze.test_table ( id INT ); INSERT INTO edw_nprd_aen.bronze.test_table VALUES (1); SELECT * FROM edw_nprd_aen.bronze.test_tab...

Dhruv-22

I want to raise a feature request as follows.Currently, in the Automatic schema evolution for merge when a column is not present in the source dataset it is not changed in the target dataset. For e.g.%sql CREATE OR REPLACE TABLE edw_nprd_aen.bronze.t...

Dhruv-22

I am referring to this link of databricks documentation. Here is a screenshot of the same According to the documentation the UPDATE command should work when the target table doesn't have the column but it is present in source. I tried the same with ...

Dhruv-22 · 10-30-2025

Today, while reading a delta load my notebook failed and I wanted to report a bug. The withColumns command does not tolerate an empty dictionary and gives the following error in PySpark.flat_tuple = namedtuple("flat_tuple", ["old_col", "new_col", "lo...

Dhruv-22 · 10-30-2025

I ran a databricks notebook to do incremental loads from files in raw layer to bronze layer tables. Today, I encountered a case where the delta file was empty. I tried running it manually on the serverless compute and encountered an error.df = spark....

Dhruv-22

Hi @ManojkMohan Thanks for pointing out the issues. I now get the pain points of writing schema dynamically. The issue with schema read while concurrent writes are happening. And in streaming / DLT tables the schema reads for each write would add to ...

Dhruv-22

Hi @css-1029 Thanks for the response, I understand the mechanics behind what is happening. But, why is it not a bug? If the the casing is to be respected then it should be across all columns and merge statements. Why does it only fail when adding a n...

Dhruv-22

Hi @ManojkMohan Can you identify the possible errors in larger pipelines? How could I achieve the required output without the above code? Your reply is not very understandable.The reason behind my feature request is as follows: The source is a NoSQL ...

Dhruv-22

Hi @iyashk-DBThanks for the response, it will help in resolving the issue.But, can you mark it as a bug and report it? Because specifying just the column without the table name is a little risky.

Dhruv-22 · 11-06-2025

hey @K_Anudeep I don't have any support subscription.

Databricks Community

User Stats

User Activity

Merge with schema evolution fails because of upper case columns

Feature request: Allow to set value as null when not present in schema evolution

BUG: Merge with schema evolution doesn't work in update clause

BUG - withColumns in pyspark doesn't handle empty dictionary

Reading empty json file in serverless gives error

Re: Feature request: Allow to set value as null when not present in schema evolution

Re: Merge with schema evolution fails because of upper case columns

Re: Feature request: Allow to set value as null when not present in schema evolution

Re: BUG: Merge with schema evolution doesn't work in update clause

Re: Reading empty json file in serverless gives error