topic Re: ImportError: cannot import name 'AnalyzeArgument' from 'pyspark.sql.udtf' in Data Engineering

ImportError: cannot import name 'AnalyzeArgument' from 'pyspark.sql.udtf'

jeremy98 — Mon, 09 Dec 2024 22:26:56 GMT

Hello community,
I installed databricks extension on my vscode ide. How to fix this error? I created the environment to run locally my notebooks and selected the available remote cluster to execute my notebook, what else?

I Have this error: ImportError: cannot import name 'AnalyzeArgument' from 'pyspark.sql.udtf'

This is the snippet code:

from databricks.connect import DatabricksSession spark = DatabricksSession.builder.getOrCreate() spark.sql("SELECT * FROM catalog.00_bronze_layer.client_email LIMIT 10")

Re: ImportError: cannot import name 'AnalyzeArgument' from 'pyspark.sql.udtf'

Alberto_Umana — Tue, 10 Dec 2024 00:39:46 GMT

Hi @jeremy98,

The error you are encountering, ImportError: cannot import name 'AnalyzeArgument' from 'pyspark.sql.udtf', is likely due to a version mismatch between the pyspark library and the databricks-connect library. This issue arises because the AnalyzeArgument class is not present in the pyspark version you are using. Could you please advise which version of pyspark and databricks-connect are you using?
Can you try: pip install --upgrade databricks-connect

Re: ImportError: cannot import name 'AnalyzeArgument' from 'pyspark.sql.udtf'

jeremy98 — Tue, 10 Dec 2024 09:44:37 GMT

Hello Alberto,
Thanks for your help. Sure, now I upgraded the Databricks-connect to v16.0.0. I was using a pyspark, but how can I find it? I was having only:

pyspark -h Python 3.13.0 (main, Oct 7 2024, 05:02:14) [Clang 16.0.0 (clang-1600.0.26.4)] on darwin Type "help", "copyright", "credits" or "license" for more information. usage: pyspark [-h] [--remote REMOTE]

In poetry should be 3.5.0

Now, I have another import error: ImportError: cannot import name 'is_remote_only' from 'pyspark.util'

Re: ImportError: cannot import name 'AnalyzeArgument' from 'pyspark.sql.udtf'

Alberto_Umana — Tue, 10 Dec 2024 13:39:07 GMT

Hi @jeremy98,

Can you also upgrade pyspark?

pip install --upgrade pyspark

Check if the is_remote_only function exists in the version of PySpark you are using. You can do this by inspecting the pyspark.util module:
```
import pyspark.util
print(dir(pyspark.util))
```

Re: ImportError: cannot import name 'AnalyzeArgument' from 'pyspark.sql.udtf'

spiky001 — Thu, 12 Dec 2024 10:19:11 GMT

Hi,

We encountered the same issues when importing sql from pyspark in the following code snippet.

from pyspark import sql def get_spark_session() -> sql.SparkSession: spark = sql.SparkSession.getActiveSession() if not spark: # trying to get a spark connect Sessions from databricks.connect import DatabricksSession from pyspark.errors.exceptions.connect import SparkConnectGrpcException spark = DatabricksSession.builder.getOrCreate() return spark

Error Encountered:

from pyspark.sql.udtf import AnalyzeArgument, AnalyzeResult # noqa: F401 ImportError: cannot import name 'AnalyzeArgument' from 'pyspark.sql.udtf'

Environment Information:

python                   3.11.10
pyspark                  3.5.0
databricks-connect       15.4.4

FYI, occasionally deleting and reinstalling the virtual environment can fix the issue, but it's not a consistent solution.

Re: ImportError: cannot import name 'AnalyzeArgument' from 'pyspark.sql.udtf'

Alberto_Umana — Fri, 13 Dec 2024 02:36:50 GMT

Hi @spiky001,

Could you please advise what is the DBR version of you cluster?

Re: ImportError: cannot import name 'AnalyzeArgument' from 'pyspark.sql.udtf'

unj1m — Thu, 19 Dec 2024 16:55:14 GMT

What version of pyspark is required? I did a clean install and got 3.5.3. I'm running Python 3.11.

$ pip freeze
cachetools==5.5.0
certifi==2024.12.14
charset-normalizer==3.4.0
databricks-connect==16.0.0
databricks-sdk==0.39.0
google-auth==2.37.0
googleapis-common-protos==1.66.0
grpcio==1.68.1
grpcio-status==1.68.1
idna==3.10
numpy==1.26.4
packaging==24.2
pandas==2.2.3
protobuf==5.29.2
py4j==0.10.9.7
pyarrow==18.1.0
pyasn1==0.6.1
pyasn1_modules==0.4.1
pyspark==3.5.3
python-dateutil==2.9.0.post0
pytz==2024.2
requests==2.32.3
rsa==4.9
six==1.17.0
tzdata==2024.2
urllib3==2.2.3

I get the error just importing databricks.connect, so I don't see how a cluster property can matter.

$ python
Python 3.11.9 (main, Aug 13 2024, 12:21:18) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import databricks.connect
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/jim/.pyenv/versions/3.11/lib/python3.11/site-packages/databricks/connect/__init__.py", line 20, in <module>
from .session import DatabricksSession
File "/home/jim/.pyenv/versions/3.11/lib/python3.11/site-packages/databricks/connect/session.py", line 28, in <module>
from .auth import DatabricksChannelBuilder
File "/home/jim/.pyenv/versions/3.11/lib/python3.11/site-packages/databricks/connect/auth.py", line 26, in <module>
from pyspark.sql.connect.client import ChannelBuilder
File "/home/jim/.pyenv/versions/3.11/lib/python3.11/site-packages/pyspark/__init__.py", line 148, in <module>
from pyspark.sql import SQLContext, HiveContext, Row # noqa: F401
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/jim/.pyenv/versions/3.11/lib/python3.11/site-packages/pyspark/sql/__init__.py", line 43, in <module>
from pyspark.sql.context import SQLContext, HiveContext, UDFRegistration, UDTFRegistration
File "/home/jim/.pyenv/versions/3.11/lib/python3.11/site-packages/pyspark/sql/context.py", line 39, in <module>
from pyspark.sql.session import _monkey_patch_RDD, SparkSession
File "/home/jim/.pyenv/versions/3.11/lib/python3.11/site-packages/pyspark/sql/session.py", line 48, in <module>
from pyspark.sql.functions import lit
File "/home/jim/.pyenv/versions/3.11/lib/python3.11/site-packages/pyspark/sql/functions/__init__.py", line 20, in <module>
from pyspark.sql.functions.builtin import * # noqa: F401,F403
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/jim/.pyenv/versions/3.11/lib/python3.11/site-packages/pyspark/sql/functions/builtin.py", line 50, in <module>
from pyspark.sql.udtf import AnalyzeArgument, AnalyzeResult # noqa: F401
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ImportError: cannot import name 'AnalyzeArgument' from 'pyspark.sql.udtf' (/home/jim/.pyenv/versions/3.11/lib/python3.11/site-packages/pyspark/sql/udtf.py)

Re: ImportError: cannot import name 'AnalyzeArgument' from 'pyspark.sql.udtf'

unj1m — Thu, 19 Dec 2024 17:31:47 GMT

I wonder if I need to install Java. 😁 I bet I do.

Re: ImportError: cannot import name 'AnalyzeArgument' from 'pyspark.sql.udtf'

Alberto_Umana — Thu, 19 Dec 2024 19:30:40 GMT

Hi @unj1m,

databricks-connect comes with pyspark “included”

Please see: https://docs.databricks.com/en/dev-tools/databricks-connect/python/install.html#install-the-databricks-connect-client-with-venv

Re: ImportError: cannot import name 'AnalyzeArgument' from 'pyspark.sql.udtf'

jeremy98 — Thu, 19 Dec 2024 19:42:57 GMT

@unj1m yes, as Alberto said you don't need to install pyspark, it is included in your cluster configuration.

Re: ImportError: cannot import name 'AnalyzeArgument' from 'pyspark.sql.udtf'

unj1m — Thu, 19 Dec 2024 23:02:12 GMT

Right you are! I actually did install pyspark and that caused the error, until I installed java.

Sorry.

Re: ImportError: cannot import name 'AnalyzeArgument' from 'pyspark.sql.udtf'

dc-rnc — Thu, 15 May 2025 15:31:10 GMT

I confirm this is the solution, it may be not consistent but it's the one that works.

Following this post steps, it fixed the issue.