Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-23-2019 11:03 AM
Hello,
i am using pyspark 2.12
After Creating Dataframe can we measure the length value for each row.
For Example: I am measuring length of a value in column 2
Input file
|TYCO|1303|
|EMC |120989|
|VOLVO|102329|
|BMW|130157|
|FORD|004|
Output in Dataframe i am trying to get like
|TYCO|1303|4
|EMC |120989|6
|VOLVO|1023295|7
|BMW|130157|6
|FORD|004| 3
Please suggest if it is possible.
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-23-2019 11:13 AM
You can use the length function for this
from pyspark.sql.functions import length
mock_data = [('TYCO', '1303'),('EMC', '120989'), ('VOLVO', '102329'),('BMW', '130157'),('FORD', '004')]
df = spark.createDataFrame(mock_data, ['col1', 'col2'])
df2 = df.withColumn('length_col2', length(df.col2))
Published notebook to show full example:
