Can't create database - UnsupportedFileSystemException No FileSystem for scheme "dbfs"

allan-silva — Wed, 23 Nov 2022 13:42:11 GMT

I'm following a class "DE 3.1 - Databases and Tables on Databricks", but it is not possible create databases due to "AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Got exception: org.apache.hadoop.fs.UnsupportedFileSystemException No FileSystem for scheme "dbfs")"

I'm able to list my files on 'dbfs://...' using `dbutils.fs.ls` utility.

Expected behavior: Successfully create database.

There are any misconfiguration ?

Detailed stacktrace:

Error in SQL statement: AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Got exception: org.apache.hadoop.fs.UnsupportedFileSystemException No FileSystem for scheme "dbfs")
com.databricks.backend.common.rpc.DatabricksExceptions$SQLExecutionException: org.apache.spark.sql.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Got exception: org.apache.hadoop.fs.UnsupportedFileSystemException No FileSystem for scheme "dbfs")
	at org.apache.spark.sql.hive.HiveExternalCatalog.$anonfun$withClient$2(HiveExternalCatalog.scala:163)
	at org.apache.spark.sql.hive.HiveExternalCatalog.maybeSynchronized(HiveExternalCatalog.scala:115)
	at org.apache.spark.sql.hive.HiveExternalCatalog.$anonfun$withClient$1(HiveExternalCatalog.scala:153)
	at com.databricks.backend.daemon.driver.ProgressReporter$.withStatusCode(ProgressReporter.scala:377)
	at com.databricks.backend.daemon.driver.ProgressReporter$.withStatusCode(ProgressReporter.scala:363)
	at com.databricks.spark.util.SparkDatabricksProgressReporter$.withStatusCode(ProgressReporter.scala:34)
	at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:152)
	at org.apache.spark.sql.hive.HiveExternalCatalog.createDatabase(HiveExternalCatalog.scala:261)
	at org.apache.spark.sql.catalyst.catalog.ExternalCatalogWithListener.createDatabase(ExternalCatalogWithListener.scala:55)
	at org.apache.spark.sql.catalyst.catalog.SessionCatalogImpl.createDatabase(SessionCatalog.scala:686)
	at com.databricks.sql.managedcatalog.ManagedCatalogSessionCatalog.createDatabase(ManagedCatalogSessionCatalog.scala:441)
	at com.databricks.sql.managedcatalog.UnityCatalogV2Proxy.$anonfun$createNamespace$1(UnityCatalogV2Proxy.scala:150)
	at com.databricks.sql.managedcatalog.UnityCatalogV2Proxy.$anonfun$createNamespace$1$adapted(UnityCatalogV2Proxy.scala:142)
	at com.databricks.sql.managedcatalog.UnityCatalogV2Proxy.assertSingleNamespace(UnityCatalogV2Proxy.scala:114)
	at com.databricks.sql.managedcatalog.UnityCatalogV2Proxy.createNamespace(UnityCatalogV2Proxy.scala:142)
	at org.apache.spark.sql.execution.datasources.v2.CreateNamespaceExec.run(CreateNamespaceExec.scala:47)
	at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.result$lzycompute(V2CommandExec.scala:43)
	at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.result(V2CommandExec.scala:43)
	at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.executeCollect(V2CommandExec.scala:49)

Re: Can't create database - UnsupportedFileSystemException No FileSystem for scheme "dbfs"

UmaMahesh1 — Wed, 23 Nov 2022 15:04:41 GMT

Can you please ping the screenshot of the cmd you are running and the error together?

Cheers...

Re: Can't create database - UnsupportedFileSystemException No FileSystem for scheme "dbfs"

allan-silva — Thu, 24 Nov 2022 08:36:27 GMT

Lab variables:

Db creation:

Thanks!

Re: Can't create database - UnsupportedFileSystemException No FileSystem for scheme "dbfs"

allan-silva — Tue, 29 Nov 2022 20:15:06 GMT

A colleague from my work figured out the problem: the cluster being used wasn't configured to use DBFS when running notebooks.

topic Re: Can't create database - UnsupportedFileSystemException No FileSystem for scheme "dbfs" in Data Engineering

Can't create database - UnsupportedFileSystemException No FileSystem for scheme "dbfs"

Re: Can't create database - UnsupportedFileSystemException No FileSystem for scheme "dbfs"

Re: Can't create database - UnsupportedFileSystemException No FileSystem for scheme "dbfs"

Re: Can't create database - UnsupportedFileSystemException No FileSystem for scheme "dbfs"