topic Re: SparkFiles - strange behavior on Azure databricks (runtime 10) in Data Engineering

SparkFiles - strange behavior on Azure databricks (runtime 10)

Hubert-Dudek — Mon, 06 Dec 2021 17:29:41 GMT

When you use:

from pyspark import SparkFiles
spark.sparkContext.addFile(url)

it adds file to NON dbfs /local_disk0/ but then when you want to read file:

spark.read.json(SparkFiles.get("file_name"))

it wants to read it from /dbfs/local_disk0/. I tried also with file:// and many other creative ways and it doesn't work.

Of course it is working after using %sh cp - moving from /local_disk0/ to /dbfs/local_disk0/ .

It seems to be a bug like addFile was switched to dbfs on azure databricks but SparkFiles not (in original spark it addFile and gets to/from workers).

I couldn't find also any settings to manually specify RootDirectory for SparkFiles.

Re: SparkFiles - strange behavior on Azure databricks (runtime 10)

weldermartins — Mon, 13 Dec 2021 13:46:55 GMT

Hello.

I'm in the same situation. Data extraction via API using sparkfiles in Community Databricks runs without error, however in Azure it generates the mentioned error.

Re: SparkFiles - strange behavior on Azure databricks (runtime 10)

jorgeff — Mon, 13 Dec 2021 19:50:21 GMT

In Azure it generates the mentioned error too

Re: SparkFiles - strange behavior on Azure databricks (runtime 10)

Hubert-Dudek — Mon, 13 Dec 2021 19:53:36 GMT

@Kaniz Fatma @Piper Wilson can you help to escalate that issue, as more people are complaining about that

Re: SparkFiles - strange behavior on Azure databricks (runtime 10)

Marcos_Gois — Mon, 13 Dec 2021 20:07:00 GMT

Hello everyone

This problem to be happening with me too, in Azure. If somebody to can help us

Re: SparkFiles - strange behavior on Azure databricks (runtime 10)

Anonymous — Mon, 13 Dec 2021 21:43:29 GMT

@Hubert Dudek - You got it!

Re: SparkFiles - strange behavior on Azure databricks (runtime 10)

weldermartins — Tue, 14 Dec 2021 11:59:57 GMT

Hi, I'm new here and I have some doubts. Will the bug fix be attended to only if there are votes, comments and views?

Re: SparkFiles - strange behavior on Azure databricks (runtime 10)

Hubert-Dudek — Tue, 14 Dec 2021 12:52:54 GMT

someone should get back to us

Re: SparkFiles - strange behavior on Azure databricks (runtime 10)

weldermartins — Tue, 14 Dec 2021 19:34:52 GMT

@Kaniz Fatma (Databricks) @Piper (Customer)

Hi how are you?

Does this problem have a solution option?

Re: SparkFiles - strange behavior on Azure databricks (runtime 10)

Hubert-Dudek — Tue, 14 Dec 2021 20:05:12 GMT

@Prabakar Ammeappin @Werner Stinckens @Jose Gonzalez maybe you could look as well to that issue 🙂

Re: SparkFiles - strange behavior on Azure databricks (runtime 10)

Anonymous — Tue, 14 Dec 2021 20:59:01 GMT

@Hubert Dudek, @Dev John, @Marcos Gois, @Jorge Fernandes, and @welder martins - Are you able to open a support ticket here - https://help.databricks.com/s/contact-us?

Re: SparkFiles - strange behavior on Azure databricks (runtime 10)

weldermartins — Tue, 14 Dec 2021 22:15:33 GMT

ok, pity it can't be solved around here. But the ticket was opened, I give news of the progress.

thanks.

Re: SparkFiles - strange behavior on Azure databricks (runtime 10)

Anonymous — Wed, 15 Dec 2021 18:54:07 GMT

@welder martins - Thank you for opening the ticket. We want to cover all our bases.

Re: SparkFiles - strange behavior on Azure databricks (runtime 10)

Hubert-Dudek — Tue, 28 Dec 2021 10:53:34 GMT

Yes this solution was already discussed on stackoverflow. Problem is that this spark functionality should be adjusted in DBR to handle everything automatically via dbfs. Problem is that it seems that it was partly adjusted but not fully.

Re: SparkFiles - strange behavior on Azure databricks (runtime 10)

weldermartins — Mon, 24 Jan 2022 11:49:11 GMT

Hello everyone, any news?

Thanks.

@Kaniz Fatma

Re: SparkFiles - strange behavior on Azure databricks (runtime 10)

weldermartins — Mon, 24 Jan 2022 12:42:49 GMT

Hi @Kaniz Fatma, Ticket Number: #00125834.

It's been over a month since the ticket was opened, but still no response.

I tested it now with version 3.2.0 of Apache Spark on the Azure platform, it continues the same way with the message: "File not found". But in community.cloud.databricks the path is found and returns the expected result.

Re: SparkFiles - strange behavior on Azure databricks (runtime 10)

weldermartins — Mon, 24 Jan 2022 13:14:38 GMT

municipios = "https://servicodados.ibge.gov.br/api/v1/localidades/municipios"
from pyspark import SparkFiles
spark.sparkContext.addFile(municipios)
 
municipiosDF = spark.read.option("multiLine", True).option("mode", "OVERRIDE").json("file://"+SparkFiles.get("municipios"))

I did not understand.

Please change the code above as instructed by you. @Kaniz Fatma

att,

Welder Martins

Re: SparkFiles - strange behavior on Azure databricks (runtime 10)

weldermartins — Mon, 24 Jan 2022 16:55:24 GMT

Hi @Kaniz Fatma (Databricks), it ran without errors. The problem is that SparkFiles doesn't work on the Azure platform. I'm extracting data from the API with other functionality. I'm even using the URLLIB function palliatively. RDD will be deprecated as of Apache Spark version 3.0.

Thak's.

Re: SparkFiles - strange behavior on Azure databricks (runtime 10)

weldermartins — Tue, 25 Jan 2022 11:39:51 GMT

@Kaniz Fatma hi, do you have access to orders that were opened in Databricks? The Ticket was opened in December 2021 and so far they have not commented on the deadline. Thanks.

Re: SparkFiles - strange behavior on Azure databricks (runtime 10)

User16764241763 — Mon, 14 Mar 2022 15:20:53 GMT

@Hubert Dudek

Have to tried with file:/// ?

I remember starting Spark 3.2, it honors the native hadoop file system if no file access protocol is defined.