Import warnings when running DBT as a package on Databricks 13.3LTS

nijhuist — Thu, 16 Nov 2023 15:05:54 GMT

Executing dbt as a Python package triggers about 200 import warnings when ran on Databricks Runtime 13.3 but not on 12.2. The warnings are all the same:

<frozen importlib._bootstrap>:914: ImportWarning: ImportHookFinder.find_spec() not found; falling back to find_module()

The warning does NOT popup when running the same python code locally either!
The warning also appear when installing dbt-core instead of dbt-databricks but since it only happens when running on Databricks clusters, I think it makes sense to post it here.

Since this 'Find_spec' warning is introduced in python 3.10, this explain why it doesn't show up on 12.2 but it does happen on 13.3.

Steps To Reproduce

Add a python file to your Databricks workspace and add this code. Run it on a Databricks cluster with 13.3LTS runtime (Shared or Single) and pre-install dbt-core 1.7.1 or dbt-databricks 1.7.1.

print("Starting") from dbt.cli.main import dbtRunner dbt = dbtRunner() print("The End")

If you want to run the script a second time, you first need to 'detach & attach' otherwise the warnings won't show up.

As a workaround, I tried suppressing the warnings using

warnings.filterwarnings(action="ignore", category=ImportWarning)

but this doesn't work. I can set the action to error (which does raise the warnings to exception) and capture it in a try...except but than the package is not imported (correctly).

So my question is: how to I get rid of these warnings?

Additional info

The original message come from this issue in dbt: [CT-2672] [Bug] Cannot import name 'Unpack' from 'typing_extensions' (dbt-core 1.5.1) · Issue #7828 · dbt-labs/dbt-core (github.com). In one of the last remarks in this thread, tlento from dbt concludes that it probably has to with the way the databricks runtime installs/clobbers its packages. That why I posted this here. Maybe there is a way to ensure the dbt package is installed after dbx runtime installed all of its packages?

I also posted the same thing in the dbt-databricks repo here: Import warnings when running DBT as a package on Databricks 13.3LTS · Issue #497 · databricks/dbt-databricks (github.com). But unfortunately it got closed without looking into it.

Re: Import warnings when running DBT as a package on Databricks 13.3LTS

nijhuist — Tue, 28 Nov 2023 08:58:55 GMT

Hi Kaniz,

Thanks for your reply!

I have tested this with Runtime 13.1, 13.3 (LTS), 13.3 (LTS) ML, 14.0, 14.1 and 14.2. All with the same result.

With 'notebook-scoped library installation' you mean doing something like this in the first cell, right?

%pip install dbt-databricks==1.7.1

I had to add a second cell with this to prevent an error:

dbutils.library.restartPython()

This second cell solved the error but the warnings are still there. Without the second cell I got the warnings as well by the way but than it ended with an import error.

I also tried adding the '--ignore-installed' flag to pip install but got the same result.

Is this what you meant by 'notebook-scoped library installation' or is there some other approach I could try?

Regards,
Thijs

topic Import warnings when running DBT as a package on Databricks 13.3LTS in Warehousing & Analytics

Import warnings when running DBT as a package on Databricks 13.3LTS

Steps To Reproduce

Additional info

Re: Import warnings when running DBT as a package on Databricks 13.3LTS