How to read gcs paths with square barkets?

Sparktaculer
New Contributor II

‎07-16-2023 07:47 AM

Hi!

 

I'm trying to read a file using Scala from gcs that has square brackets in the file path.

I keep getting the following error:URISyntaxException: Illegal character in path at index 209

I tried putting an extra front slash in front of them but it still didn't work.

Would really appreciate you're help ere!

0 Kudos

Sparktaculer
New Contributor II
In response to Retired_mod

‎07-19-2023 01:47 AM - edited ‎07-19-2023 01:53 AM

Hi @Retired_mod ! Thank you for your help.

However, when I tray using you're code I still get an error : "URISyntaxException: Illegal character in path at index "

I'm trying to read a txt file. This is the file path: 

"gs://my-bucket/my Data/sparkTests/GM-1220, reading a txt/Version1/3 Model Creation/3 models_to_check/[no_country] (2)/test.txt"
 
This is how I'm trying to read the file:
def loadFromGCS(gcsUrl: String😞 (String, Boolean, RecordClassifier) = {
  val content = spark.sparkContext.textFile(gcsUrl).collect().mkString("\n")
  print(content)}

 

0 Kudos

Tharun-Kumar
Databricks Employee
Databricks Employee

‎07-18-2023 04:34 AM

Hi @Sparktaculer 

In Spark, you can disable the option globPaths. This will skip the pattern matching that happens during file reads.

spark.read.option("__globPaths__", False).format("").load("path[]")

 

Anonymous
Not applicable

‎07-19-2023 03:27 AM

Hi @Sparktaculer, 

We haven't heard from you since the last response from @Tharun-Kumar and @Retired_mod , and I was checking back to see if her suggestions helped you.

Or else, If you have any solution, please share it with the community, as it can be helpful to others. 

Also, Please don't forget to click on the "Select As Best" button whenever the information provided helps resolve your question.

0 Kudos

Sparktaculer
New Contributor II

‎07-19-2023 03:50 AM

Hi @Kaniz ! Thank you for your help.

However, when I try using you're code I still get an error: "URISyntaxException: Illegal character in path at index "

I'm trying to read a txt file. This is the file path: 

"gs://my-bucket/my Data/sparkTests/GM-1220, reading a txt/Version1/3 Model Creation/3 models_to_check/[no_country] (2)/test.txt"
This is how I'm trying to read the file:
def loadFromGCS(gcsUrl: String
 (String, Boolean, RecordClassifier) = {
  val content = spark.sparkContext.textFile(gcsUrl).collect().mkString("\n")
  print(content)}
0 Kudos