My experience with Databricks pyspark up to this point has always been to execute a SQL query against existing Databricks tables, then write the resulting pyspark dataframe into a new table. 

For the first time, I am now getting data via an API which returns a large XML blob containing a table of data. I convert the XML object to a BeautifulSoup object and extract the table data into a Pandas data frame. That takes a couple of minutes, not so bad. Then I convert the Pandas data frame to a pyspark data frame. That takes about a minute. Then I issue a command to write that pyspark dataframe to a table, and it takes... TWO HOURS.

Now, the dataframe has about 130K rows with 39 columns. Two hours to write 130K rows? What is going on?

By contrast, yesterday I developed a pyspark script saving a dataframe created from a pyspark SQL command. The dataframe had about 180K rows, and the entire job ran in six minutes. 

Why does a pyspark dataframe created from a pandas dataframe created from a beautifulsoup object take two hours to write to a table, while a similar sized pyspark dataframe created from a pyspark query takes only six minutes from running the query all the way to writing the table?