https://community.databricks.com/t5/data-engineering/how-to-process-files-from-the-internet-in-databricks-quot-spark/m-p/24443#M16986 <P>@Hubert Dudek​&nbsp;Do you know if addFile should work with abfss:// path? Trying to add a file from azure data lake with external location in unity catalog. </P> Tue, 10 Jan 2023 22:24:36 GMT Matt101122 2023-01-10T22:24:36Z https://community.databricks.com/t5/data-engineering/how-to-process-files-from-the-internet-in-databricks-quot-spark/m-p/24441#M16984 <P>How to process files from the internet in databricks?</P><P></P><P>"spark.sparkContext.addFile" download file to HDFS directory. "SparkFiles.get" return the path and the name. However, as Databricks use the DBFS file system, we need to add the "file:///" prefix to get files from the legacy HDFS system.</P><P></P><P><span class="lia-inline-image-display-wrapper" image-alt="image.png"><img src="https://community.databricks.com/t5/image/serverpage/image-id/1987i3D839DC55B94E0B5/image-size/large?v=v2&amp;px=999" role="button" title="image.png" alt="image.png" /></span>&nbsp;</P> Mon, 28 Mar 2022 18:29:41 GMT https://community.databricks.com/t5/data-engineering/how-to-process-files-from-the-internet-in-databricks-quot-spark/m-p/24441#M16984 Hubert-Dudek 2022-03-28T18:29:41Z https://community.databricks.com/t5/data-engineering/how-to-process-files-from-the-internet-in-databricks-quot-spark/m-p/24443#M16986 <P>@Hubert Dudek​&nbsp;Do you know if addFile should work with abfss:// path? Trying to add a file from azure data lake with external location in unity catalog. </P> Tue, 10 Jan 2023 22:24:36 GMT https://community.databricks.com/t5/data-engineering/how-to-process-files-from-the-internet-in-databricks-quot-spark/m-p/24443#M16986 Matt101122 2023-01-10T22:24:36Z