ImportError: cannot import name 'AnalyzeArgument' from 'pyspark.sql.udtf'

[jeremy98]

Hello community,
I installed databricks extension on my vscode ide. How to fix this error? I created the environment to run locally my notebooks and selected the available remote cluster to execute my notebook, what else?

I Have this error: ImportError: cannot import name 'AnalyzeArgument' from 'pyspark.sql.udtf'

This is the snippet code:

from databricks.connect import DatabricksSession
spark = DatabricksSession.builder.getOrCreate()

spark.sql("SELECT * FROM catalog.00_bronze_layer.client_email LIMIT 10")

[Alberto_Umana]

Hi @jeremy98,

The error you are encountering, ImportError: cannot import name 'AnalyzeArgument' from 'pyspark.sql.udtf', is likely due to a version mismatch between the pyspark library and the databricks-connect library. This issue arises because the AnalyzeArgument class is not present in the pyspark version you are using. Could you please advise which version of pyspark and databricks-connect are you using?
Can you try: pip install --upgrade databricks-connect

[jeremy98]

Hello Alberto,
Thanks for your help. Sure, now I upgraded the Databricks-connect to v16.0.0. I was using a pyspark, but how can I find it? I was having only:

pyspark -h                
Python 3.13.0 (main, Oct  7 2024, 05:02:14) [Clang 16.0.0 (clang-1600.0.26.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
usage: pyspark [-h] [--remote REMOTE]

In poetry should be 3.5.0

Now, I have another import error: ImportError: cannot import name 'is_remote_only' from 'pyspark.util'

[Alberto_Umana]

Hi @jeremy98,

Can you also upgrade pyspark?

pip install --upgrade pyspark

• Check if the is_remote_only function exists in the version of PySpark you are using. You can do this by inspecting the pyspark.util module:

  import pyspark.util
  print(dir(pyspark.util))