Perfect time for this to happen -- right in the middle of a class assignment due in a few days. 🙂
The easiest way to do it is by using dbutils.fs.mv to mv the file from the local system to the dbfs. Here is a complete example:
## create file in the local fs
with open("./test_file.txt", "w") as file:
lines = "\n".join(map(str, range(5)))
file.write(lines)
!cat test_file.txt
>> 0 1 2 3 4
#mv file from local fs to dbf
dbutils.fs.mv("file:/databricks/driver/test_file.txt", "dbfs:/test_file.txt")
>> Out[10]: True
# Read the file as an RDD
file_path = "dbfs:/test_file.txt"
rdd = sc.textFile(file_path)
rdd.collect()
>> Out[11]: ['0', '1', '2', '3', '4']
