cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

SaraCorralLou
by New Contributor III
  • 14416 Views
  • 2 replies
  • 0 kudos

Resolved! Differences between lit(None) or lit(None).cast('string')

I want to define a column with null values in my dataframe using pyspark. This column will later be used for other calculations.What is the difference between creating it in these two different ways?df.withColumn("New_Column", lit(None))df.withColumn...

  • 14416 Views
  • 2 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Sara Corral​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feedback ...

  • 0 kudos
1 More Replies
cfregly
by Contributor
  • 5498 Views
  • 5 replies
  • 0 kudos
  • 5498 Views
  • 5 replies
  • 0 kudos
Latest Reply
srisre111
New Contributor II
  • 0 kudos

I am trying to store a dataframe as table in databricks and encountering the following error, can someone help? "typeerror: field date: can not merge type <class 'pyspark.sql.types.stringtype'> and <class 'pyspark.sql.types.doubletype'>"

  • 0 kudos
4 More Replies
dan11
by New Contributor II
  • 4029 Views
  • 1 replies
  • 1 kudos

sql: how to convert datatype of column?

Bricklayers, I want to port this sql statement from sqlite to databricks: select cast(myage as number) as my_integer_age from ages; Does databricks allow me to do something like this?

  • 4029 Views
  • 1 replies
  • 1 kudos
Latest Reply
raela
Databricks Employee
  • 1 kudos

@dan11 We don't support number in Spark SQL. Try using int, double, float, and your query should be fine. To run SQL in a notebook, just prepend any cell with %sql. %sql select cast(myage as double) as my_integer_age from ages;

  • 1 kudos
Labels