Dynamically detect if any dataframe column is an array type, to perform logic on that column

ilarsen — Fri, 13 Oct 2023 03:54:58 GMT

Hi,

I'd like to put this out here in case there are some helpful suggestions to be found.

What am I trying to achieve?

Generate a hash of certain columns in a dataframe (as in a row hash, but not the whole row) where currently one of the columns is an array of struct. Without explicitly referencing the column(s) by name.

Where have I got to?

I have achieved what I want to do... sort of. By specifying the columns and using the sha2() and to_json() functions to convert the array of structs in to a string, enabling me to use sha2.

What's the problem?

I don't want to specify the column (or columns) by name. The data is coming from an API, it's JSON format, and I want to safeguard against changes in schema. If the API payload changes without warning, my aim is for our process to adjust without intervention. So, if the current array of nested objects column changes name, I don't want it to break.

What have I tried?
I've tried playing around with dataframe.schema and dataframe.dtypes. I can't get a simple conditional true/false return if the column is an array. The datatypes seem to be ArrayType, yes, but following from that, listing the schema, so all the nested columns etc. So I haven't got something like if dataType is array: true else false working.

Source format example:

{ "name":"value", "Array" [ { "Id":1234 "Name":"some name" ... }, { ... } ] }

Anyone have any ideas?

Re: Dynamically detect if any dataframe column is an array type, to perform logic on that column

-werners- — Fri, 13 Oct 2023 07:34:33 GMT

That is totally possible.
f.e. here is a function that trims all string columns in a dataframe. You can change it to your needs:

def trim_all_string_columns(df: dataframe) -> dataframe: for c in df.schema.fields: if isinstance(c.dataType, StringType): df = df.withColumn(c.name, F.trim(F.col(c.name))) return df

Re: Dynamically detect if any dataframe column is an array type, to perform logic on that column

ilarsen — Wed, 25 Oct 2023 22:31:25 GMT

Thanks for that. The isinstance is what I was looking for and did help me out. Although, I didn't end up continuing on that track.

topic Re: Dynamically detect if any dataframe column is an array type, to perform logic on that column in Data Engineering

Dynamically detect if any dataframe column is an array type, to perform logic on that column

Re: Dynamically detect if any dataframe column is an array type, to perform logic on that column

Re: Dynamically detect if any dataframe column is an array type, to perform logic on that column