cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

how can I pass the df columns as a parameter

g96g
New Contributor III

Im doing the self study and want pass df column name as a parameter.

I have defined the widget

column_name= dbutils.widgets.get('column_name')

which is executing succefuly ( giving me a column name)

then Im reading the df and do some transformation and after that I want to wrtie back to datalake.

try:
    (df2[column_name].write
     .mode('append')
     .parquet(f'/mnt/bro/Default/{table_name}'))
except Exception as e:
    print(e)
    raise Exception(e)

then, idea is that adf pipeline will loop through different files after.

but when i run the above code I get error

'DataFrame' object has no attribute 'column_name'

Im sorry if my explanation is not good enough, I just started learning.

1 ACCEPTED SOLUTION

Accepted Solutions

Hubert-Dudek
Esteemed Contributor III
df2.select([column_name]).write

OR

df2.select(column_name).write

View solution in original post

1 REPLY 1

Hubert-Dudek
Esteemed Contributor III
df2.select([column_name]).write

OR

df2.select(column_name).write

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group