Sorry if this notifies everyone again - struggling with the text editor here!

I'm not sure of a workaround for directly splitting with more than one character. 2 ways around this that I can see:

The first approach would be to split using "|" and then replace the angle brackets:

dff = sqlContext.read.format("com.databricks.spark.csv").option("header", "true").option("inferSchema", "true").option("delimiter", "|").load(trainingdata+"part-00000")

from pyspark.sql.functions import regexp_replace dff = dff.withColumn('pageId', regexp_replace('[pageId]','[[]]',''))....other columns....

Given that you have 50 columns you might want to loop through this:

dffs_headers = dff.dtypes
for i in dffs_headers: 
    newColumnLabel = i[0].replace('[','').replace(']','') 
    dff = dff.withColumn(newColumnLabel, regexp_replace(i[0],'[[]]','')).drop(i[0]) 

You'll still need to build a function or add cases to this loop to correctly cast each column though. The second way would be go via RDD:

dff = sc.textfile(trainingdata+"part-00000").map(lambda x: x.replace('[','').replace(']','').split('|')).toDF() 

But that still leaves you with the casting problem Hope that helps!