topic Generate sh2 hashkey while loading files to delta table in Data Engineering

Generate sh2 hashkey while loading files to delta table

Vijaykumarj — Thu, 05 Jan 2023 05:23:52 GMT

I have files in azure data lake. I am using autoloader to read the incremental files

files don't have primary key to load, In this case i want to use some columns and generate an hashkey and use it as primary key to do changes.

In this case i want to load my initial file with haskkey column should be appended

and for microbatches also hashkey needs to be appended .

but while i am using sh2 to generate hash key getting error

inp file1:

inputpath = 'abfss://***@***.dfs.core.windows.net/test/'

df = spark.readStream.format("cloudFiles").option("cloudFiles.format","csv").option("cloudFiles.schemaEvolutionMode","rescue").option("cloudFIles.schemaLocation", checkpoint_path).load(inputpath)

df.withColumns("Hashkey",sha2(concat_ws(",",df['id'],df['product_Name'],df['Location'],df['offer_code']),256))

getting

AssertionError:

Re: Generate sh2 hashkey while loading files to delta table

Hubert-Dudek — Thu, 05 Jan 2023 10:15:46 GMT

Can you copy the whole error?

I bet that it should be withColumn not with withColumns (remove s)

Re: Generate sh2 hashkey while loading files to delta table

Vijaykumarj — Fri, 06 Jan 2023 02:15:00 GMT

Sorry for delay in response, refer below for error

Re: Generate sh2 hashkey while loading files to delta table

Jfoxyyc — Fri, 06 Jan 2023 06:04:52 GMT

Try withColumn. withColumns takes an array of columns and does something with them, like rename using regex. withColumn creates new column named whatever.

Re: Generate sh2 hashkey while loading files to delta table

Debayan — Thu, 05 Jan 2023 18:43:13 GMT

Hi , Could you please provide the error code?