Pass Databricks's Spark session to a user defined module
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-19-2023 01:50 PM
Hello everyone,
I have a .py file (not a notebook) where I have the following class with the following constructor:
class DataQualityChecker:
def __init__(self, spark_session: SparkSession, df: DataFrame, quality_config_filepath: str) -> None:
self.__spark = spark_session
self.__df = df
self.__quality_config = self.__import_quality_config(quality_config_filepath)
self.__error_constraints = NoneI'm trying to use this class in a notebook as follows:
from data_quality import DataQualityChecker
quality_checker = DataQualityChecker(spark, df, quality_config)However, I am receiving the following error in the code snippet above:
TypeError: __init__() takes 3 positional arguments but 4 were givenDoes anyone have any idea what it could be? I suspect the problem is trying to pass the spark variable as a parameter to the class constructor, but I don't know how to solve it.