topic Re: java.lang.IllegalArgumentException: java.net.URISyntaxException in Data Engineering

java.lang.IllegalArgumentException: java.net.URISyntaxException

Prem1 — Wed, 10 Aug 2022 22:00:57 GMT

I am using Databricks Autoloader to load JSON files from ADLS gen2 incrementally in directory listing mode. All source filename has Timestamp on them. The autoloader works perfectly couple of days with the below configuration and breaks the next day with the following error.

org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 7.0 failed 4 times, most recent failure: Lost task 1.3 in stage 7.0 (TID 24) (10.150.38.137 executor 0): java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: 2022-04-27T20:09:00 (Attached the complete error message)

I deleted the checkpoint, and target delta table and loaded fresh with the option "cloudFiles.includeExistingFiles":"true". All files loaded successfully and then after a couple of incremental loads the same error occurred.

Autoloader configurations

{"cloudFiles.format":"json","cloudFiles.useNotifications":"false", "cloudFiles.inferColumnTypes":"true", "cloudFiles.schemaEvolutionMode":"addNewColumns", "cloudFiles.includeExistingFiles":"false"}

Path location passed as below

raw_data_location : dbfs:/mnt/DEV-cdl-raw/data/storage-xxxxx/xxxx/

target_delta_table_location : dbfs:/mnt/DEV-cdl-bronze/data/storage-xxxxx/xxxx/

checkpoint_location : dbfs:/mnt/DEV-cdl-bronze/configuration/autoloader/storage-xxxxx/xxxx/checkpoint/

schema_location : dbfs:/mnt/DEV-cdl-bronze/metadata/storage-xxxxx/xxxx/

StreamingQuery = StreamDF.writeStream \

.option("checkpointLocation", checkpoint_location) \

.option("mergeSchema", "true") \

.queryName(f"AutoLoad_RawtoBronze_{sourceFolderName}_{sourceEntityName}") \

.trigger(availableNow=True) \

.partitionBy(targetPartitionByCol) \

.start(target_delta_table_location)

Can someone help me here?

Thanks in advance.

Re: java.lang.IllegalArgumentException: java.net.URISyntaxException

Alexey — Wed, 17 Aug 2022 12:06:51 GMT

I think you are running into the same problem as I do right now: Autoloader (or something even deeper) doesn't like ":" (colon) in the file names. 😕

Loading the files with the simple Spark read option works fine.

Re: java.lang.IllegalArgumentException: java.net.URISyntaxException

Prem1 — Thu, 18 Aug 2022 03:44:27 GMT

I don't understand the part why it's not consistent with the failure. It runs fine for a few runs then it stops with this error. It is a strange situation.

Re: java.lang.IllegalArgumentException: java.net.URISyntaxException

Alexey — Thu, 18 Aug 2022 07:37:59 GMT

so, for me it breaks directly on the first file with the colon in the name.

Re: java.lang.IllegalArgumentException: java.net.URISyntaxException

Vidula — Thu, 08 Sep 2022 11:23:47 GMT

Hi there @PREM KUMAR KUMMAN RAMESH

Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help.

We'd love to hear from you.

Thanks!

Re: java.lang.IllegalArgumentException: java.net.URISyntaxException

Alexey — Fri, 09 Sep 2022 13:39:35 GMT

Hi @Vidula Khanna

I can report from my side: I wasn't able to solve the issue with the AutoLoader. For my daily job, I first perform

os.walk(...)

in Python and check if there are any files with "colon" in the naming (and some other criteria). If everything is fine, I use AutoLoader for incremental load, else I reload the data every time.

Luckily for us, the biggest chunk of data is fine, but I hope that this issue will be fixed some day.

Re: java.lang.IllegalArgumentException: java.net.URISyntaxException

B_Seibert — Mon, 21 Nov 2022 13:53:11 GMT

Yes, for us it runs several times before error.

Re: java.lang.IllegalArgumentException: java.net.URISyntaxException

B_Seibert — Mon, 21 Nov 2022 13:56:03 GMT

@Alexey Egorov , can you tell us more precisely what you mean by "I reload the data"?

Re: java.lang.IllegalArgumentException: java.net.URISyntaxException

B_Seibert — Mon, 21 Nov 2022 14:05:34 GMT

Previously we have run AutoLoader many times on very similar folder names without a fail. Now we get:

StreamingQueryException: Job aborted due to stage failure: Task 1 in stage 1657.0 failed 4 times, most recent failure: Lost task 1.3 in stage 1657.0 (TID 5451) (10.38.20.138 executor 17): java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: 2022-03-07T20:47:0

"2022-03-07T20.47.04.000Z" = fails now in November
"2022-03-07T20.47.04.000Z" = succeeded from July through October

DESCRIBE HISTORY

operation (most recent at top)

MERGE

ADD COLUMNS --started having problems after this

RESTORE

MERGE

Re: java.lang.IllegalArgumentException: java.net.URISyntaxException

Alexey — Mon, 21 Nov 2022 14:33:37 GMT

By reloading I mean to load all the existing data in that folder. As mentioned above:

if there are no special charaters that make AutoLoader fail we can do:

autoloader = spark.readStream.format("cloudFiles") \

.option("cloudFiles.format", data_format) \

.option("header", "true") \

.option("cloudFiles.schemaLocation", schema_location) \

.option("cloudFiles.allowOverwrites", "true") \

.load(path)

in the second case, where Autloader will fail (at least we know from experience, that it does with the colon in the file names), we use simple data load:

df = spark.read.format(data_format)\

.option("header", "true") \

.load(path)

That is why I mentioned that luckily for us, this data folder is not that huge and it works fast.

Re: java.lang.IllegalArgumentException: java.net.URISyntaxException

Andrei_Radulesc — Mon, 21 Nov 2022 15:20:33 GMT

For me, the simple read also throws an "java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI" exception when it encounters files with ':' in the name:

df1= spark.read.format("binaryFile").load("s3://bucket_name/*/*/*/*.bag")

So the problem is not solved.

Re: java.lang.IllegalArgumentException: java.net.URISyntaxException

Alexey — Mon, 21 Nov 2022 15:26:16 GMT

wait, but I think this is another problem. We are mounting an S3 bucket into DBFS and my path is then something like this:

S3_BUCKET_PATH = "dbfs:/mnt/mounted_bucket_name/"

df = spark.read.format(format).load(S3_BUCKET_PATH)

Re: java.lang.IllegalArgumentException: java.net.URISyntaxException

Andrei_Radulesc — Mon, 21 Nov 2022 15:52:49 GMT

Here is the stacktrace:

Re: java.lang.IllegalArgumentException: java.net.URISyntaxException

B_Seibert — Mon, 21 Nov 2022 16:10:07 GMT

Have either of you .changed the schema? We experienced the problem after we did ADD COLUMNS

Re: java.lang.IllegalArgumentException: java.net.URISyntaxException

B_Seibert — Mon, 21 Nov 2022 17:04:56 GMT

We did not get an error until we ran ADD COLUMNS. Has anyone else done similar DDL changes? BTW I get the same error in Azure.

Re: java.lang.IllegalArgumentException: java.net.URISyntaxException

Andrei_Radulesc — Mon, 21 Nov 2022 18:03:41 GMT

I did not change the schema. The schema is fixed in my case.

Re: java.lang.IllegalArgumentException: java.net.URISyntaxException

Andrei_Radulesc — Mon, 21 Nov 2022 22:14:09 GMT

For me, it's the same error when mounting through a mount point:

schema = StructType() \

.add("path", StringType(), False) \

.add("modificationTime", StringType(), False) \

.add("length", IntegerType(), False) \

.add("content", BinaryType(), True)

df = spark.read.format("binaryFile").schema(schema) \

.load("dbfs:/mnt/bucket_name/[...]/*/*.bag")

IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: 2022-11-16T22:01:49+00:00

Surely enough, I have some files in there with a ':' character in the name. Incidentally, the Databricks architect who advised us a while back said that mount points are obsolete, and don't play well with the Unity Catalog permission scheme, so I've tried to refrain from using mount points.

Re: java.lang.IllegalArgumentException: java.net.URISyntaxException

Prem1 — Tue, 03 Jan 2023 15:17:24 GMT

Hello to everyone. We filed a support ticket with Databricks. This is the response I received, along with an interim solution to the problem. I hope it is useful to those who read it.

Problem Statement:

Root Cause Analysis:

We have an incremental listing mode to speedup the listing by not scanning prefixes we saw before but the incremental listing mode does not like file names with certain special char in it for example:

and if you upload a file with a special character into DBFS file gets renamed and automatically replaced by _

This is currently in our roadmap but don't have an exact ETA.

Solution:

use the below config to mitigate the issue

cloudFiles.useIncrementalListing to false.

Re: java.lang.IllegalArgumentException: java.net.URISyntaxException

jshields — Wed, 04 Jan 2023 14:56:35 GMT

Hi Everyone,

I'm seeing this issue as well - same configuration of the previous posts, using autoloader with incremental file listing turned on. The strange part is that it mostly works despite almost all of the files we're loading having colons included as part of the timestamp.

It seems to be happening more frequently now, which is becoming an issue. Having to rarely clear a checkpoint is very different than needing to clear checkpoints each day. I'm also not comfortable clearing the checkpoint programmatically. I'm lucky in our case that I have some level of control of how the files get named, so removing colons from the timestamp is a possibility.

Including a couple bullet points here for anyone else struggling though this.

This issue is intermittent - I haven't seen any pattern as to what causes this to occur with certain files.
This is using Databricks in Azure using adlsg2. - streaming cluster is Databricks runtime 10.4LTS
When an autoloader task fails due to this issue, it is broken permanently until the checkpoint gets cleared.

I haven't tried setting cloudFiles.useIncrementalListing to false, but this also feels like an unideal fix for my purposes. I'll be following updates in this tread closely, Also thanks for everyone who already shared info previously.

Re: java.lang.IllegalArgumentException: java.net.URISyntaxException

Alexey — Mon, 16 Jan 2023 12:59:58 GMT

Thanks for sharing your experience. Will the setting for useIncrementalListing somehow change your processing strategy? I think it is only for performance improvements, isn't it? I am really not quite sure. But seems actually more like a bug, that will be fixed one day. It would be great if someone from databricks would comment here when this is fixed.