cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

How can I create from scratch a brand new Dataframe with Null values using spark.createDataFrame()?

User15787040559
New Contributor III
from pyspark.sql.types import *
schema = StructType([
  StructField("c1", IntegerType(), True),
  StructField("c2", StringType(), True),
  StructField("c3", StringType(), True)])
df = spark.createDataFrame([(1, "2", None), (3, "4", None)], schema)

1 REPLY 1

Mooune_DBU
Valued Contributor
df = spark.createDataFrame(sc.emptyRDD(), schema)

Can you try this?

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.