cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

how can I pass the df columns as a parameter

g96g
New Contributor III

Im doing the self study and want pass df column name as a parameter.

I have defined the widget

column_name= dbutils.widgets.get('column_name')

which is executing succefuly ( giving me a column name)

then Im reading the df and do some transformation and after that I want to wrtie back to datalake.

try:
    (df2[column_name].write
     .mode('append')
     .parquet(f'/mnt/bro/Default/{table_name}'))
except Exception as e:
    print(e)
    raise Exception(e)

then, idea is that adf pipeline will loop through different files after.

but when i run the above code I get error

'DataFrame' object has no attribute 'column_name'

Im sorry if my explanation is not good enough, I just started learning.

1 ACCEPTED SOLUTION

Accepted Solutions

Hubert-Dudek
Esteemed Contributor III
df2.select([column_name]).write

OR

df2.select(column_name).write

View solution in original post

2 REPLIES 2

Hubert-Dudek
Esteemed Contributor III
df2.select([column_name]).write

OR

df2.select(column_name).write

Kaniz
Community Manager
Community Manager

Hi @Givi Salu​ , We haven’t heard from you since the last response from @Hubert Dudek​, and I was checking back to see if you have a resolution yet.

If you have any solution, please share it with the community as it can be helpful to others. Otherwise, we will respond with more details and try to help.

Also, Please don't forget to click on the "Select As Best" button whenever the information provided helps resolve your question.

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.