topic Re: How do I read the contents of a hidden file in a Spark job? in Data Engineering

How do I read the contents of a hidden file in a Spark job?

Lincoln_Bergeso — Tue, 15 Feb 2022 23:24:33 GMT

I'm trying to read a file from a Google Cloud Storage bucket. The filename starts with a period, so Spark assumes the file is hidden and won't let me read it.

My code is similar to this:

from pyspark.sql import SparkSession
 
spark = SparkSession.builder.getOrCreate()
df = spark.read.format("text").load("gs://<bucket>/.myfile", wholetext=True)
df.show()

The resulting DataFrame is empty (as in, it has no rows).

When I run this on my laptop, I get the following error message:

22/02/15 16:40:58 WARN DataSource: All paths were ignored:
  gs://<bucket>/.myfile

I've noticed that this applies to files starting with an underscore as well.

How can I get around this?

Re: How do I read the contents of a hidden file in a Spark job?

-werners- — Wed, 16 Feb 2022 16:00:56 GMT

Spark uses the Hadoop Input API to read files, which ignores every file that starts with an underscore or a period.

I did not find a solution for this as the hiddenFileFilter is always active.

Re: How do I read the contents of a hidden file in a Spark job?

Anonymous — Wed, 16 Feb 2022 16:18:53 GMT

Hi there, @Lincoln Bergeson! My name is Piper, and I'm a moderator for Databricks. Thank you for your question and welcome to the community. We'll give your peers a chance to respond and then we'll circle back if we need to.

Thanks in advance for your patience. 🙂

Re: How do I read the contents of a hidden file in a Spark job?

Lincoln_Bergeso — Wed, 16 Feb 2022 17:13:06 GMT

Is there any way to work around this?

Re: How do I read the contents of a hidden file in a Spark job?

Lincoln_Bergeso — Wed, 16 Feb 2022 17:13:40 GMT

Looking forward to the answers. From my research this looks something that needs a special configuration or work-around, which I'm hoping Databricks can provide.

Re: How do I read the contents of a hidden file in a Spark job?

Atanu — Wed, 16 Mar 2022 04:34:10 GMT

@Lincoln Bergeson GCS object names are very liberal. Only \r and \n are invalid, everything else is valid, including the NUL character. I am still not sure if this can help you. We do really need to hack this from spark side!

Re: How do I read the contents of a hidden file in a Spark job?

jose_gonzalez — Mon, 11 Apr 2022 18:51:20 GMT

Hi @Lincoln Bergeson ,

Just a friendly follow-up. Did any of the previous responses help you to resolve your issue? Please let us know if you still need help.

Re: How do I read the contents of a hidden file in a Spark job?

Lincoln_Bergeso — Tue, 12 Apr 2022 18:49:18 GMT

Hi @Jose Gonzalez , none of these answers helped me, unfortunately. I'm still hoping to find a good solution to this issue.

Re: How do I read the contents of a hidden file in a Spark job?

Dan_Z — Wed, 04 May 2022 16:19:30 GMT

I don't think there is an easy way to do this. You will also break very basic functionality (like being able to read Delta tables) if you were able to get around these constraints. I suggest you employ a rename job and then read.