Import warnings when running DBT as a package on Databricks 13.3LTS
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-16-2023 07:04 AM - edited 11-16-2023 07:05 AM
Executing dbt as a Python package triggers about 200 import warnings when ran on Databricks Runtime 13.3 but not on 12.2. The warnings are all the same:
<frozen importlib._bootstrap>:914: ImportWarning: ImportHookFinder.find_spec() not found; falling back to find_module()
The warning does NOT popup when running the same python code locally either!
The warning also appear when installing dbt-core instead of dbt-databricks but since it only happens when running on Databricks clusters, I think it makes sense to post it here.
Since this 'Find_spec' warning is introduced in python 3.10, this explain why it doesn't show up on 12.2 but it does happen on 13.3.
Steps To Reproduce
Add a python file to your Databricks workspace and add this code. Run it on a Databricks cluster with 13.3LTS runtime (Shared or Single) and pre-install dbt-core 1.7.1 or dbt-databricks 1.7.1.
print("Starting")
from dbt.cli.main import dbtRunner
dbt = dbtRunner()
print("The End")
If you want to run the script a second time, you first need to 'detach & attach' otherwise the warnings won't show up.
As a workaround, I tried suppressing the warnings using
warnings.filterwarnings(action="ignore", category=ImportWarning)
but this doesn't work. I can set the action to error (which does raise the warnings to exception) and capture it in a try...except but than the package is not imported (correctly).
So my question is: how to I get rid of these warnings?
Additional info
The original message come from this issue in dbt: [CT-2672] [Bug] Cannot import name 'Unpack' from 'typing_extensions' (dbt-core 1.5.1) · Issue #7828 .... In one of the last remarks in this thread, tlento from dbt concludes that it probably has to with the way the databricks runtime installs/clobbers its packages. That why I posted this here. Maybe there is a way to ensure the dbt package is installed after dbx runtime installed all of its packages?
I also posted the same thing in the dbt-databricks repo here: Import warnings when running DBT as a package on Databricks 13.3LTS · Issue #497 · databricks/dbt-da.... But unfortunately it got closed without looking into it.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-28-2023 12:58 AM
Hi Kaniz,
Thanks for your reply!
I have tested this with Runtime 13.1, 13.3 (LTS), 13.3 (LTS) ML, 14.0, 14.1 and 14.2. All with the same result.
With 'notebook-scoped library installation' you mean doing something like this in the first cell, right?
%pip install dbt-databricks==1.7.1
I had to add a second cell with this to prevent an error:
dbutils.library.restartPython()
This second cell solved the error but the warnings are still there. Without the second cell I got the warnings as well by the way but than it ended with an import error.
I also tried adding the '--ignore-installed' flag to pip install but got the same result.
Is this what you meant by 'notebook-scoped library installation' or is there some other approach I could try?
Regards,
Thijs

