Databricks Community

Dhruv-22

Today, while reading a delta load my notebook failed and I wanted to report a bug. The withColumns command does not tolerate an empty dictionary and gives the following error in PySpark.flat_tuple = namedtuple("flat_tuple", ["old_col", "new_col", "lo...

Dhruv-22

I ran a databricks notebook to do incremental loads from files in raw layer to bronze layer tables. Today, I encountered a case where the delta file was empty. I tried running it manually on the serverless compute and encountered an error.df = spark....

Dhruv-22

I have a pipeline which puts json files in a storage location after reading a daily delta load. Today I encountered a case where the file as empty. I tried running the notebook manually using serverless cluster (Environment version 4) and encountered...

Dhruv-22

I have a table which has a column of data type 'bigint'. While overwriting it with new data, given that I do full loads, I used 'mergeSchema' to handle schema changes. The new data's datatype was int. I thought mergeSchema can easily handle that, but...

Dhruv-22

The following statement gives different outputs in different computes.In Databricks, 15.4 LTS%sqlSELECT typeof(coalesce(5, '6'));-- OutputstringIn Serverless, environment version 4%sqlSELECT typeof(coalesce(5, '6'));-- OutputbigintThere are other cas...

Dhruv-22

hey @K_Anudeep I don't have any support subscription.

Dhruv-22

Adding to my point. Suppose the file consists of a valid json along with the byte order mark like below (the bytes ef, bb and bf represent the byte order mark)dhruv@AS-MAC-0324 Downloads % cat zone.json|hexdump -C00000000 ef bb bf 7b 22 61 22 3a 2...

Dhruv-22

Hi @K_Anudeep I searched more. It is not a corrupt record. The three characters represent a Byte Order Mark (BOM) signaling that the file is UTF-8 encoded. It is a standard thing. Also, the file is generated automatically by a no-code pipeline (Azure...

Dhruv-22

Hey @K_Anudeep, did you get anything?

Dhruv-22

Hey @K_AnudeepHere are the details you requested.Yes, there is a _corrupt_record column in the dataframe. It is coming since spark is treating the file to have some corrupt records. Therefore, it is generating the _corrupt_record columnThe error come...

Databricks Community

User Stats

User Activity

BUG - withColumns in pyspark doesn't handle empty dictionary

Reading empty json file in serverless gives error

Reading empty json file in serverless gives error

Can't mergeSchema handle int and bigint?

Least Common Type is different in Serverless and All Purpose Cluster.

Re: Reading empty json file in serverless gives error

Re: Reading empty json file in serverless gives error

Re: Reading empty json file in serverless gives error

Re: BUG - withColumns in pyspark doesn't handle empty dictionary

Re: Reading empty json file in serverless gives error