topic Re: Bug in db.fs.utils in Data Engineering

Bug in db.fs.utils

740209 — Tue, 27 Sep 2022 19:42:12 GMT

When using db.fs.utils on a s3 bucket titled "${sometext}.${sometext}.${somenumber}${sometext}-${sometext}-${sometext}" we receive an error. PLEASE understand this is an issue with how it encodes the .${somenumber} because we verified with boto3 that we can query the s3 bucket. There is already a community issue that I believe is due to this, but does not provide this amount of detail. https://community.databricks.com/s/question/0D58Y00009BMsKGSA1/dbutilsfsls-is-giving-null-uri-host-this-can-be-caused-by-unencoded-in-the-password-string

Command (works with other buckets that do not follow this format, but doesn't work on any bucket that follows format above):

dbutils.fs.ls("s3a://${bucket_name_here_follow_above_format}")

Traceback:

ExecutionError: An error occurred while calling z:com.databricks.backend.daemon.dbutils.FSUtils.ls. : java.lang.NullPointerException: null uri host. This can be caused by unencoded / in the password string at java.util.Objects.requireNonNull(Objects.java:228) at shaded.databricks.org.apache.hadoop.fs.s3native.S3xLoginHelper.buildFSURI(S3xLoginHelper.java:69) at shaded.databricks.org.apache.hadoop.fs.s3a.S3AFileSystem.setUri(S3AFileSystem.java:792) at shaded.databricks.org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:406) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3469)

Re: Bug in db.fs.utils

740209 — Mon, 03 Oct 2022 16:43:40 GMT

@Debayan Mukherjee

All the information is there please read accurately. I am not going to give you the actual bucket name I am using on a public forum. As i said above here is the command:

dbutils.fs.ls("s3a://${bucket_name_here_follow_above_format}")

where format is "${sometext}.${sometext}.${somenumber}${sometext}-${sometext}-${sometext}"

so an example command is: dbutils.fs.ls("s3a://testtext.testtext.4303testtext-testtext-testtext").

We had to use an S3 access point name because the actual name of bucket did not work and has the above Traceback.

Re: Bug in db.fs.utils

740209 — Tue, 18 Oct 2022 14:00:33 GMT

We can't submit a support case apparently because this requires a support contract. Our databricks contact told us to submit a community request.

Re: Bug in db.fs.utils

Debayan — Fri, 30 Sep 2022 06:30:27 GMT

Hi, Could you please type in the whole command and enter? (sometimes copying will put the format incorrectly).

Reference: https://docs.databricks.com/data/data-sources/aws/amazon-s3.html

Re: Bug in db.fs.utils

Debayan — Thu, 06 Oct 2022 06:51:06 GMT

Hi @arturo.iacobelli arturo.iacobelli , Thanks for the information. We appreciate your patience in this regard! I would request you to file a support case in Databricks portal with all the details so that we can look into the issue thoroughly and triage the same.