Hi,

We encountered the same issues when importing sql from pyspark in the following code snippet.

from pyspark import sql

def get_spark_session() -> sql.SparkSession:
    spark = sql.SparkSession.getActiveSession()
    if not spark:
        # trying to get a spark connect Sessions
        from databricks.connect import DatabricksSession
        from pyspark.errors.exceptions.connect import SparkConnectGrpcException
        spark = DatabricksSession.builder.getOrCreate()
    return spark

Error Encountered:

from pyspark.sql.udtf import AnalyzeArgument, AnalyzeResult  # noqa: F401
ImportError: cannot import name 'AnalyzeArgument' from 'pyspark.sql.udtf'

Environment Information:

python                   3.11.10
pyspark                  3.5.0
databricks-connect       15.4.4

FYI, occasionally deleting and reinstalling the virtual environment can fix the issue, but it's not a consistent solution.