https://community.databricks.com/t5/data-engineering/pyspark-pandas-pandasnotimplementederror/m-p/62138#M31907 <P><a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/2427">@pjp94</a>&nbsp; - The error indicates the pandas pyspark implementation does not have the below method implemented.</P> <PRE class="lia-code-sample language-markup"><CODE>pd.Series.duplicated()</CODE></PRE> <P>Next steps is to use dataframe methods such as distinct, groupBy, dropDuplicates to resolve this.</P> Tue, 27 Feb 2024 17:47:22 GMT shan_chandra 2024-02-27T17:47:22Z https://community.databricks.com/t5/data-engineering/pyspark-pandas-pandasnotimplementederror/m-p/61621#M31815 <P>Can someone explain why this below code is throwing an error? My intuition is telling me it's my spark version (3.2.1) but would like confirmation:</P><LI-CODE lang="markup">d = {'key':['a','a','c','d','e','f','g','h'], 'data':[1,2,3,4,5,6,7,8]} x = ps.DataFrame(d) x[x['key'].duplicated()] ----------------------------------------------------------------------------- PandasNotImplementedError Traceback (most recent call last) &lt;command-6518367&gt; in &lt;module&gt; 3 4 x = ps.DataFrame(d) ----&gt; 5 x[x['key'].duplicated()] /databricks/spark/python/pyspark/pandas/usage_logging/__init__.py in wrapper(*args, **kwargs) 192 start = time.perf_counter() 193 try: --&gt; 194 res = func(*args, **kwargs) 195 logger.log_success( 196 class_name, function_name, time.perf_counter() - start, signature /databricks/spark/python/pyspark/pandas/series.py in __getattr__(self, item) 6276 property_or_func = getattr(MissingPandasLikeSeries, item) 6277 if isinstance(property_or_func, property): -&gt; 6278 return property_or_func.fget(self) # type: ignore 6279 else: 6280 return partial(property_or_func, self) /databricks/spark/python/pyspark/pandas/usage_logging/__init__.py in wrapper(self) 261 def wrapper(self): 262 try: --&gt; 263 return prop.fget(self) 264 finally: 265 logger.log_missing(class_name, property_name, is_deprecated) /databricks/spark/python/pyspark/pandas/missing/__init__.py in unsupported_property(self) 36 @property 37 def unsupported_property(self): ---&gt; 38 raise PandasNotImplementedError( 39 class_name=class_name, property_name=property_name, reason=reason 40 ) PandasNotImplementedError: The property `pd.Series.duplicated()` is not implemented. 'duplicated' API returns np.ndarray and the data size is too large.You can just use DataFrame.deduplicated instead</LI-CODE><P>&nbsp;</P> Thu, 22 Feb 2024 22:37:11 GMT https://community.databricks.com/t5/data-engineering/pyspark-pandas-pandasnotimplementederror/m-p/61621#M31815 pjp94 2024-02-22T22:37:11Z https://community.databricks.com/t5/data-engineering/pyspark-pandas-pandasnotimplementederror/m-p/62138#M31907 <P><a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/2427">@pjp94</a>&nbsp; - The error indicates the pandas pyspark implementation does not have the below method implemented.</P> <PRE class="lia-code-sample language-markup"><CODE>pd.Series.duplicated()</CODE></PRE> <P>Next steps is to use dataframe methods such as distinct, groupBy, dropDuplicates to resolve this.</P> Tue, 27 Feb 2024 17:47:22 GMT https://community.databricks.com/t5/data-engineering/pyspark-pandas-pandasnotimplementederror/m-p/62138#M31907 shan_chandra 2024-02-27T17:47:22Z