cancel
Showing results for 
Search instead for 
Did you mean: 
Databricks Free Trial Help
Engage in discussions about the Databricks Free Trial within the Databricks Community. Share insights, tips, and best practices for getting started, troubleshooting issues, and maximizing the value of your trial experience to explore Databricks' capabilities effectively.
cancel
Showing results for 
Search instead for 
Did you mean: 

How to check databricks-connect types of objects.

DiegoMX
New Contributor

While using `databricks-sdk` in my code, I've found that checking PySpark objects types is not reliable anymore.  
I've used to do the following: 

 

from pyspark.sql import Column, DataFrame, SparkSession

isinstance(spark, SparkSession)
isinstance(a_df, DataFrame)
isinstance(a_col, Column)

 

And so I've found that this isn't reliable as it dependes on the context in which I'm running Spark.  
That is to say: 

 

type(spark) -> pyspark.sql.(connect).session.SparkSession
type(a_df)  -> pyspark.sql.(connect).dataframe.DataFrame
type(a_col) -> pyspark.sql.(connect).column.Column

 

Where depending on the context Databricks may add it's `connect` module to the object in question.  
Am I not supposed to check types when using PySpark in Databricks? 
Is there a reference for this change that I can follow? 
And more practically, how do I check if the dataframe is a dataframe? 
The only way I can think of at the moment is: 

 

from pyspark.sql import SparkSession, Column, DataFrame
from pyspark.sql.connect import session, dataframe, column

isinstance(spark, (SparkSession, session.SparkSession))
isinstance(a_df,  (DataFrame, dataframe.DataFrame))
isinstance(a_col, (Column, column.Column))

 

 Is there a more natural way to do this type of checking?  

0 REPLIES 0

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group