How to process files from the internet in databricks? "spark.sparkContext.addFile" download file to HDFS directory. "SparkFiles.get&quo...

Hubert-Dudek
Databricks MVP

‎03-28-2022 11:29 AM

How to process files from the internet in databricks?

"spark.sparkContext.addFile" download file to HDFS directory. "SparkFiles.get" return the path and the name. However, as Databricks use the DBFS file system, we need to add the "file:///" prefix to get files from the legacy HDFS system.

image.png 


My blog: https://databrickster.medium.com/
Labels:
Preview file
153 KB

Matt101122
Contributor II

‎01-10-2023 02:24 PM

@Hubert Dudek​ Do you know if addFile should work with abfss:// path? Trying to add a file from azure data lake with external location in unity catalog.

0 Kudos