How to process files from the internet in databricks? "spark.sparkContext.addFile" download file to HDFS directory. "SparkFiles.get&quo...

[Hubert-Dudek]

How to process files from the internet in databricks?

"spark.sparkContext.addFile" download file to HDFS directory. "SparkFiles.get" return the path and the name. However, as Databricks use the DBFS file system, we need to add the "file:///" prefix to get files from the legacy HDFS system.

 

[Kaniz]

Awesome post.

Thank you @Hubert Dudek​ .

[Matt101122]

@Hubert Dudek​ Do you know if addFile should work with abfss:// path? Trying to add a file from azure data lake with external location in unity catalog.