Difference between Databricks and local pyspark split.
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-14-2023 12:13 AM
I have noticed some inconsistent behavior between calling the 'split' fuction on databricks and on my local installation.
Running it in a databricks notebook gives
spark.sql("SELECT split('abc', ''), size(split('abc',''))").show()So the string is split in 3 parts.
If I run on my local install I get 4 parts.
Locally I'm running on pyspark 3.2.1, on databricks I've tried it with multiple versions but they all give the same result.