Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-17-2024 01:24 AM - edited 09-17-2024 01:37 AM
Hitting the same problems trying to check the type of variables to pick out DataFrames.
Ended up getting around this (temporarily at least) by importing the following instead:
from pyspark.sql.connect.dataframe import DataFrame
both 'isinstance(df, DataFrame)' then works again for my dataframe variables that are of type 'pyspark.sql.connect.dataframe.DataFrame'
(if you have already imported from 'from pyspark.sql import DataFrame' you probably need to 'del DataFrame' then redo the import above).
Note that this does however produce a console message as follows, so ymmv:
sc will be removed in future DBR versions
sc will be removed in future DBR versions
