topic Re: Unsupported cloud file system schema 'wasbs' error in custom catalog, but not in hive_metastore in Data Engineering

Unsupported cloud file system schema 'wasbs' error in custom catalog, but not in hive_metastore

DianGermishuiz1 — Wed, 15 Feb 2023 16:43:16 GMT

When trying to run a create table USING CSV that pulls data from an azure blob storage path in a custom catalog I created, I get an error stating Unsupported cloud file system schema 'wasbs'.

However, when I run this code in the hive_metastore catalog, it runs fine.

What permissions do I need to set on my custom catalog to allow the code to succeed?

The blob storage path is provided in the Databricks SQL Analyst training course - https://partner-academy.databricks.com/learn/course/1035/play/5293/getting-started-course-setup;lp=78

Image of the error:

Re: Unsupported cloud file system schema 'wasbs' error in custom catalog, but not in hive_metastore

DianGermishuiz1 — Thu, 16 Feb 2023 05:46:23 GMT

Hello,

Tried changing the path, however i get a new error

"No parent external location found for path 'abfs://courseware@dbacademy.dfs.core.windows.net/data-analysis-with-databricks/v01/flights/departuredelays.csv'"

Is it perhaps that the files are still stored in a standard blob storage and not in a Gen2 Data Lake ?

Re: Unsupported cloud file system schema 'wasbs' error in custom catalog, but not in hive_metastore

Anonymous — Tue, 21 Feb 2023 10:02:59 GMT

Hi @Dian Germishuizen

Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help.

We'd love to hear from you.

Thanks!

Re: Unsupported cloud file system schema 'wasbs' error in custom catalog, but not in hive_metastore

DianGermishuiz1 — Tue, 21 Feb 2023 14:06:07 GMT

Hi @Vidula Khanna ,

No resolution as yet.

Tried changing the path, however i get a new error

"No parent external location found for path 'abfs://courseware@dbacademy.dfs.core.windows.net/data-analysis-with-databricks/v01/flights/departuredelays.csv'"

Is it perhaps that the files are still stored in a standard blob storage and not in a Gen2 Data Lake ?

Re: Unsupported cloud file system schema 'wasbs' error in custom catalog, but not in hive_metastore

Hemant — Thu, 23 Feb 2023 03:58:09 GMT

Hello @Dian Germishuizen,

It appears that the wasbs or abfss protocol is not functioning properly in the SQL editor. However, you can still utilize these protocols to create a mount point in the data engineer persona using a notebook. Once the mount point is created, you can use it in the path option to create the table.

I hope this helps! Let me know if you have any further questions.

Re: Unsupported cloud file system schema 'wasbs' error in custom catalog, but not in hive_metastore

youssefmrini — Thu, 23 Feb 2023 09:34:17 GMT

WASB Has been deprecated.

Re: Unsupported cloud file system schema 'wasbs' error in custom catalog, but not in hive_metastore

DianGermishuiz1 — Fri, 24 Feb 2023 06:03:25 GMT

Hi Hemant,

Thank you for the reply. I will try this out and see how it goes.

@Vidula Khanna can you please pass this message on to Databricks Product / Support team to update the SQL Analyst learning path training to indicate that you can only do it in the default hive_metastore catalog, not a custom catalog.

Also to just fix whatever is causing the problem in custom catalogs.

Thank you

Re: Unsupported cloud file system schema 'wasbs' error in custom catalog, but not in hive_metastore

Debayan — Thu, 16 Feb 2023 05:16:31 GMT

Hi, this has to be abfss:// (The legacy Windows Azure Storage Blob driver (WASB) has been deprecated. ABFS has numerous benefits over WASB; see Azure documentation on ABFS. For documentation for working with the legacy WASB driver, see Connect to Azure Blob Storage with WASB (legacy).)