Understanding Rename in Databricks Now there are multiple ways to rename Spark Data Frame Columns or Expressions. We can rename columns or expressions...

Aviral-Bhardwaj — Sun, 29 Jan 2023 06:16:45 GMT

Understanding Rename in Databricks

Now there are multiple ways to rename Spark Data Frame Columns or Expressions.

We can rename columns or expressions using alias as part of select

We can add or rename columns or expressions using withColumn on top of the Data Frame.

We can rename one column at a time using withColumnRenamed on top of the Data Frame.

We typically use withColumn to perform row-level transformations and then to provide a name to the result. If we give the same name as an existing column, then the column will be replaced with a new one.

If we want just to rename the column then it is better to use withColumnRenamed.

If we want to apply any transformation, we need to use select or withColumn

We can rename a bunch of columns using toDF.

First, we will understand what is the withColumnRenamed and withColumn

withColumn

help(MyFirstDataFrame.withColumn)
 
#RESULT
 
Help on method withColumn in module pyspark.sql.dataframe:
 
withColumn(colName: str, col: pyspark.sql.column.Column) -> 'DataFrame' method of pyspark.sql.dataframe.DataFrame instance
    Returns a new :class:`DataFrame` by adding a column or replacing the
    existing column that has the same name.
    
    The column expression must be an expression over this :class:`DataFrame`; attempting to add
    a column from some other :class:`DataFrame` will raise an error.
    
    .. versionadded:: 1.3.0
    
    Parameters
    ----------
    colName : str
        string, name of the new column.
    col : :class:`Column`
        a :class:`Column` expression for the new column.
    
    Notes
    -----
    This method introduces a projection internally. Therefore, calling it multiple
    times, for instance, via loops in order to add multiple columns can generate big
    plans which can cause performance issues and even `StackOverflowException`.
    To avoid this, use :func:`select` with the multiple columns at once.
    
    Examples
    --------
    >>> df.withColumn('age2', df.age + 2).collect()
    [Row(age=2, name='Alice', age2=4), Row(age=5, name='Bob', age2=7)]

withColumnRenamed

help(MyFirstDataFrame.withColumnRenamed)
 
#RESULT
 
Help on method withColumnRenamed in module pyspark.sql.dataframe:
 
withColumnRenamed(existing: str, new: str) -> 'DataFrame' method of pyspark.sql.dataframe.DataFrame instance
    Returns a new :class:`DataFrame` by renaming an existing column.
    This is a no-op if schema doesn't contain the given column name.
    
    .. versionadded:: 1.3.0
    
    Parameters
    ----------
    existing : str
        string, name of the existing column to rename.
    new : str
        string, new name of the column.
    
    Examples
    --------
    >>> df.withColumnRenamed('age', 'age2').collect()
    [Row(age2=2, name='Alice'), Row(age2=5, name='Bob')]

Now we will do its Practical in our Dataframe

withColumn -We can create whole New Column

eg:withColumn(NewColumn, OldColumn)

MyFirstDataFrame.select('FirstName','LastName','age','city','Salary','Bonus') \
.withColumn('NewColFirstName',MyFirstDataFrame.FirstName).show()
 
#RESULT
 
+---------+--------+---+-----------+------+-----+---------------+
|FirstName|LastName|age|       city|Salary|Bonus|NewColFirstName|
+---------+--------+---+-----------+------+-----+---------------+
|      Ram|  Tiwari| 25|  Bangalore|  2000|  100|            Ram|
|    Shyam|      NC| 30|    Chennai|  3000|  300|          Shyam|
|    Rohan|    Jaki| 45|     Andhra|  1500|  150|          Rohan|
|   Ritesh|  sharma| 35|Rameshwaram|  2500|  250|         Ritesh|
+---------+--------+---+-----------+------+-----+---------------+
 
#I HAVE ADDED FORWARD SLASH for just visbility of code you can remove that
 
#Now using the []
 
MyFirstDataFrame.select('FirstName','LastName','age','city','Salary','Bonus').\
withColumn('NewColFirstName',MyFirstDataFrame['FirstName']).show()
 
#Result
+---------+--------+---+-----------+------+-----+---------------+
|FirstName|LastName|age|       city|Salary|Bonus|NewColFirstName|
+---------+--------+---+-----------+------+-----+---------------+
|      Ram|  Tiwari| 25|  Bangalore|  2000|  100|            Ram|
|    Shyam|      NC| 30|    Chennai|  3000|  300|          Shyam|
|    Rohan|    Jaki| 45|     Andhra|  1500|  150|          Rohan|
|   Ritesh|  sharma| 35|Rameshwaram|  2500|  250|         Ritesh|
+---------+--------+---+-----------+------+-----+---------------+
 
#Now i hope you understood what I am trying to say

withColumnRenamed- We can create whole New Column eg:withColumnRenamed(OldColumnName, NewName)

#our Old Result
MyFirstDataFrame.select("*").show()
 
+---------+--------+---+-----------+------+-----+
|FirstName|LastName|age|       city|Salary|Bonus|
+---------+--------+---+-----------+------+-----+
|      Ram|  Tiwari| 25|  Bangalore|  2000|  100|
|    Shyam|      NC| 30|    Chennai|  3000|  300|
|    Rohan|    Jaki| 45|     Andhra|  1500|  150|
|   Ritesh|  sharma| 35|Rameshwaram|  2500|  250|
+---------+--------+---+-----------+------+-----+
 
 
MyFirstDataFrame.select('FirstName','LastName','age','city','Salary','Bonus')./
withColumnRenamed('FirstName', 'UsersFirstName')./
withColumnRenamed('LastName', 'userLastName')./
withColumnRenamed('age', 'UserAge').show()
 
#Result
 
+--------------+------------+-------+-----------+------+-----+
|UsersFirstName|userLastName|UserAge|       city|Salary|Bonus|
+--------------+------------+-------+-----------+------+-----+
|           Ram|      Tiwari|     25|  Bangalore|  2000|  100|
|         Shyam|          NC|     30|    Chennai|  3000|  300|
|         Rohan|        Jaki|     45|     Andhra|  1500|  150|
|        Ritesh|      sharma|     35|Rameshwaram|  2500|  250|
+--------------+------------+-------+-----------+------+-----+

and Finally, we have an alias function for changing the Column name below is the code

#our Old Result
MyFirstDataFrame.select("*").show()
 
+---------+--------+---+-----------+------+-----+
|FirstName|LastName|age|       city|Salary|Bonus|
+---------+--------+---+-----------+------+-----+
|      Ram|  Tiwari| 25|  Bangalore|  2000|  100|
|    Shyam|      NC| 30|    Chennai|  3000|  300|
|    Rohan|    Jaki| 45|     Andhra|  1500|  150|
|   Ritesh|  sharma| 35|Rameshwaram|  2500|  250|
+---------+--------+---+-----------+------+-----+
 
MyFirstDataFrame. \
    select(
        MyFirstDataFrame['FirstName'].alias('UsersFirstName'),
        MyFirstDataFrame['LastName'].alias('UserLastName'),
        MyFirstDataFrame['age'].alias('UserAge')
    ). \
    withColumn('User_Full_Name', concat(col('UsersFirstName'), lit(', '), col('userLastName'))). \
    show()
 
#Result
 
+--------------+------------+-------+--------------+
|UsersFirstName|UserLastName|UserAge|User_Full_Name|
+--------------+------------+-------+--------------+
|           Ram|      Tiwari|     25|   Ram, Tiwari|
|         Shyam|          NC|     30|     Shyam, NC|
|         Rohan|        Jaki|     45|   Rohan, Jaki|
|        Ritesh|      sharma|     35|Ritesh, sharma|
+--------------+------------+-------+--------------+

Re: Understanding Rename in Databricks Now there are multiple ways to rename Spark Data Frame Columns or Expressions. We can rename columns or expressions...

Ajay-Pandey — Mon, 30 Jan 2023 12:17:21 GMT

Very informative, Thanks for sharing

Re: Understanding Rename in Databricks Now there are multiple ways to rename Spark Data Frame Columns or Expressions. We can rename columns or expressions...

Aviral-Bhardwaj — Mon, 30 Jan 2023 14:26:12 GMT

thank you sir

topic Re: Understanding Rename in Databricks Now there are multiple ways to rename Spark Data Frame Columns or Expressions. We can rename columns or expressions... in Data Engineering

Understanding Rename in Databricks Now there are multiple ways to rename Spark Data Frame Columns or Expressions. We can rename columns or expressions...

Re: Understanding Rename in Databricks Now there are multiple ways to rename Spark Data Frame Columns or Expressions. We can rename columns or expressions...

Re: Understanding Rename in Databricks Now there are multiple ways to rename Spark Data Frame Columns or Expressions. We can rename columns or expressions...