how can I pass the df columns as a parameter

g96g — Wed, 19 Oct 2022 15:27:34 GMT

Im doing the self study and want pass df column name as a parameter.

I have defined the widget

column_name= dbutils.widgets.get('column_name')

which is executing succefuly ( giving me a column name)

then Im reading the df and do some transformation and after that I want to wrtie back to datalake.

try:
    (df2[column_name].write
     .mode('append')
     .parquet(f'/mnt/bro/Default/{table_name}'))
except Exception as e:
    print(e)
    raise Exception(e)

then, idea is that adf pipeline will loop through different files after.

but when i run the above code I get error

'DataFrame' object has no attribute 'column_name'

Im sorry if my explanation is not good enough, I just started learning.

Re: how can I pass the df columns as a parameter

Hubert-Dudek — Thu, 20 Oct 2022 14:53:21 GMT

df2.select([column_name]).write

df2.select(column_name).write

topic Re: how can I pass the df columns as a parameter in Data Engineering

how can I pass the df columns as a parameter

Re: how can I pass the df columns as a parameter