Bug in db.fs.utils
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-27-2022 12:42 PM
When using db.fs.utils on a s3 bucket titled "${sometext}.${sometext}.${somenumber}${sometext}-${sometext}-${sometext}" we receive an error. PLEASE understand this is an issue with how it encodes the .${somenumber} because we verified with boto3 that we can query the s3 bucket. There is already a community issue that I believe is due to this, but does not provide this amount of detail. https://community.databricks.com/s/question/0D58Y00009BMsKGSA1/dbutilsfsls-is-giving-null-uri-host-t...
Command (works with other buckets that do not follow this format, but doesn't work on any bucket that follows format above):
dbutils.fs.ls("s3a://${bucket_name_here_follow_above_format}")
Traceback:
ExecutionError: An error occurred while calling z:com.databricks.backend.daemon.dbutils.FSUtils.ls. : java.lang.NullPointerException: null uri host. This can be caused by unencoded / in the password string at java.util.Objects.requireNonNull(Objects.java:228) at shaded.databricks.org.apache.hadoop.fs.s3native.S3xLoginHelper.buildFSURI(S3xLoginHelper.java:69) at shaded.databricks.org.apache.hadoop.fs.s3a.S3AFileSystem.setUri(S3AFileSystem.java:792) at shaded.databricks.org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:406) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3469)