To close out this thread we found the issue we were having with serverless didn't have to do with our import, but with the isinstance check we are using in our if statements for different functions. In short, serverless uses a different DataFrame type than a compute cluster behind the scenes. This issue is mentioned in the below article:

pyspark.sql.connect.dataframe.DataFrame vs pyspark... - Databricks Community - 71055