cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
PrasadGaikwad
by • New Contributor
  • 11800 Views
  • 1 replies
  • 0 kudos

TypeError: Column is not iterable when using more than one columns in withColumn()

I am trying to find quarter start date from a date column. I get the expected result when i write it using selectExpr() but when i add the same logic in .withColumn() i get TypeError: Column is not iterableI am using a workaround as follows workarou...

Data Engineering
Reply
  • 11800 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
emma_s
Databricks Employee
  • 0 kudos

Hi, This is a super old question but answering in case anyone else comes across it. This isn't working because the add months expects an integer rather than a column name you can get round this by using expr() inside withColumn: from pyspark.sql.func...

  • 0 kudos
pramalin
by • New Contributor
  • 4565 Views
  • 3 replies
  • 2 kudos

How to perform Inner join using withcolumn

Data Engineering
Reply
  • 4565 Views
  • 3 replies
  • 2 kudos
View Replies
Latest Reply
shan_chandra
Databricks Employee
  • 2 kudos

@prudhvi ramalingam​ - Please refer to the below example code.import org.apache.spark.sql.functions.expr val person = Seq( (0, "Bill Chambers", 0, Seq(100)), (1, "Matei Zaharia", 1, Seq(500, 250, 100)), (2, "Michael Armbrust", 1, Seq(250,...

  • 2 kudos
2 More Replies
weldermartins
by • Honored Contributor
  • 5245 Views
  • 3 replies
  • 6 kudos

Resolved! Function When + Dictionary.

Hey everyone, I'm avoiding repeating the When Function for 12x, so I thought of the dictionary. I don't know if it's a limitation of the Spark function or a Logic error. Does the function allow this concatenation?

image
Data Engineering
Reply
  • 5245 Views
  • 3 replies
  • 6 kudos
View Replies
Latest Reply
weldermartins
Honored Contributor
  • 6 kudos

Hello everyone, I found this alternative to reduce repeated code.custoDF = (custoDF.withColumn('month', col('Nummes').cast('string')) .replace(months, subset=['month']))

  • 6 kudos
2 More Replies
Labels