cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
SaraCorralLou
by • New Contributor III
  • 15402 Views
  • 3 replies
  • 2 kudos

Resolved! Differences between lit(None) or lit(None).cast('string')

I want to define a column with null values in my dataframe using pyspark. This column will later be used for other calculations.What is the difference between creating it in these two different ways?df.withColumn("New_Column", lit(None))df.withColumn...

Data Engineering
Reply
  • 15402 Views
  • 3 replies
  • 2 kudos
View Replies
Latest Reply
shadowinc
New Contributor III
  • 2 kudos

For me df.withColumn("New_Column", lit(None).cast(StringType())) this didn't work.I used this instead df.withColumn("New_Column", lit(null).cast(StringType))  

  • 2 kudos
2 More Replies
cfregly
by • Contributor
  • 5762 Views
  • 5 replies
  • 0 kudos

How do I cast using a DataFrame?

Data Engineering
Reply
  • 5762 Views
  • 5 replies
  • 0 kudos
View Replies
Latest Reply
srisre111
New Contributor II
  • 0 kudos

I am trying to store a dataframe as table in databricks and encountering the following error, can someone help? "typeerror: field date: can not merge type <class 'pyspark.sql.types.stringtype'> and <class 'pyspark.sql.types.doubletype'>"

  • 0 kudos
4 More Replies
dan11
by • New Contributor II
  • 4103 Views
  • 1 replies
  • 1 kudos

sql: how to convert datatype of column?

Bricklayers, I want to port this sql statement from sqlite to databricks: select cast(myage as number) as my_integer_age from ages; Does databricks allow me to do something like this?

Data Engineering
Reply
  • 4103 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
raela
Databricks Employee
  • 1 kudos

@dan11 We don't support number in Spark SQL. Try using int, double, float, and your query should be fine. To run SQL in a notebook, just prepend any cell with %sql. %sql select cast(myage as double) as my_integer_age from ages;

  • 1 kudos
Labels