topic Re: Database Error in model Couldn't initialize file system for path abfss:// in Administration & Architecture

Database Error in model Couldn't initialize file system for path abfss://

Th0r — Tue, 17 Dec 2024 06:26:33 GMT

Recently the following error ocurs when running DBT:

Database Error in model un_unternehmen_sat (models/2_un/partner/sats/un_unternehmen_sat.sql)
Couldn't initialize file system for path abfss://dp-ext-fab@stcssdpextfabprd.dfs.core.windows.net/__unitystorage/catalogs/64552de9-4ff0-4c5f-b4a7-0aef0d6c821a/tables/1bd27775-ca51-435a-8c1a-1042391ac471/part-00000-8121312f-d7a0-4a5b-964a-4c2f4ea4f2ec-c000.snappy.parquet

Has anybody an idea how to fix this error or how to further analyze it?

Re: Database Error in model Couldn't initialize file system for path abfss://

filipniziol — Tue, 17 Dec 2024 07:23:58 GMT

Hi @Th0r ,

The error typically occurs due to one of the following reasons:

The specified file or path does not exist.
The credentials or configurations for accessing the storage account are misconfigured.

Could you confirm if the following file exists in your storage:

abfss://dp-ext-fab@stcssdpextfabprd.dfs.core.windows.net/__unitystorage/catalogs/64552de9-4ff0-4c5f-b4a7-0aef0d6c821a/tables/1bd27775-ca51-435a-8c1a-1042391ac471/part-00000-8121312f-d7a0-4a5b-964a-4c2f4ea4f2ec-c000.snappy.parquet

Re: Database Error in model Couldn't initialize file system for path abfss://

Th0r — Tue, 17 Dec 2024 08:38:21 GMT

The error occured becase this table is a shallow clone from another table. The original table was re-created.

To fix this we dropped the table and made another shallow clone.

Re: Database Error in model Couldn't initialize file system for path abfss://

Th0r — Tue, 17 Dec 2024 08:40:34 GMT

Yeah the file is not there - but the question is why / how to fix it.

It was due the original source of a shallow clone was re-created.

So we made the shallow clone again to fix it.

Thx for pointing out some hints

Re: Database Error in model Couldn't initialize file system for path abfss://

filipniziol — Tue, 17 Dec 2024 08:52:00 GMT

Hi @Th0r ,

Here is the explanation:

Shallow clones in Databricks rely on references to data files of the original table. If the original table is dropped, recreated, or altered in a way that changes its underlying files, the shallow clone’s references become stale.
When dbt attempts to read these stale references, it fails because the referenced files are no longer present in the storage.

Solutions:

Recreate the Shallow Clone: (what you have done) Since shallow clones depend on the original table’s files, if the source has been altered and the files are not in place, you can recreate the shallow clone.

Use a Full Clone (if applicable): If the original table is changed frequently, consider using a full clone to retain independent physical copies of the data. This avoids breakages when the source is altered