Import warnings when running DBT as a package on Databricks 13.3LTS

[nijhuist]

Executing dbt as a Python package triggers about 200 import warnings when ran on Databricks Runtime 13.3 but not on 12.2. The warnings are all the same: 

<frozen importlib._bootstrap>:914: ImportWarning: ImportHookFinder.find_spec() not found; falling back to find_module()

The warning does NOT popup when running the same python code locally either!
The warning also appear when installing dbt-core instead of dbt-databricks but since it only happens when running on Databricks clusters, I think it makes sense to post it here.

Since this 'Find_spec' warning is introduced in python 3.10, this explain why it doesn't show up on 12.2 but it does happen on 13.3.

Steps To Reproduce

Add a python file to your Databricks workspace and add this code. Run it on a Databricks cluster with 13.3LTS runtime (Shared or Single) and pre-install dbt-core 1.7.1 or dbt-databricks 1.7.1.

print("Starting")

from dbt.cli.main import dbtRunner

dbt = dbtRunner()

print("The End")

If you want to run the script a second time, you first need to 'detach & attach' otherwise the warnings won't show up.

As a workaround, I tried suppressing the warnings using 

warnings.filterwarnings(action="ignore", category=ImportWarning)

 but this doesn't work. I can set the action to error (which does raise the warnings to exception) and capture it in a try...except but than the package is not imported (correctly).

So my question is: how to I get rid of these warnings?

Additional info

The original message come from this issue in dbt: [CT-2672] [Bug] Cannot import name 'Unpack' from 'typing_extensions' (dbt-core 1.5.1) · Issue #7828 .... In one of the last remarks in this thread, tlento from dbt concludes that it probably has to with the way the databricks runtime installs/clobbers its packages. That why I posted this here. Maybe there is a way to ensure the dbt package is installed after dbx runtime installed all of its packages?

I also posted the same thing in the dbt-databricks repo here: Import warnings when running DBT as a package on Databricks 13.3LTS · Issue #497 · databricks/dbt-da.... But unfortunately it got closed without looking into it. 

[Kaniz_Fatma]

Hi @nijhuist , Note: These are potential approaches; please test them in your specific environment.

• Cluster Version:
  • Create your own cluster with the suggested runtime (if possible). 
  • Sometimes using a different cluster version can resolve such issues.
• Notebook-Scoped Library Installation:
  • Use notebook-scoped library installation commands within your Databricks Notebook.

1.  

Feel free to explore these solutions, and I hope they help you mitigate those pesky warnings! 🚀

[nijhuist]

Hi Kaniz,

Thanks for your reply!

I have tested this with Runtime 13.1, 13.3 (LTS),  13.3 (LTS) ML, 14.0, 14.1 and 14.2. All with the same result. 

With 'notebook-scoped library installation' you mean doing something like this in the first cell, right? 

%pip install dbt-databricks==1.7.1

I had to add a second cell with this to prevent an error:

dbutils.library.restartPython()

This second cell solved the error but the warnings are still there. Without the second cell I got the warnings as well by the way but than it ended with an import error.

I also tried adding the '--ignore-installed' flag to pip install but got the same result.

Is this what you meant by 'notebook-scoped library installation' or is there some other approach I could try?

Regards,
Thijs