topic Re: Problem with dropDuplicates in Databricks runtime 15.4LTS in Data Engineering

Problem with dropDuplicates in Databricks runtime 15.4LTS

juan_barreto — Thu, 12 Sep 2024 15:03:06 GMT

Hi,
I'm testing the latest version of the databricks runtime but I'm getting errors doing a simple dropDuplicates.

Using the following code

data = spark.read.table("some_table") data.dropDuplicates(subset=['SOME_COLUMN']).count()

I'm getting this error.

TypeError Traceback (most recent call last) File <command-934417477504931>, line 1 ----> 1 data.dropDuplicates(subset=['SOME_COLUMN']).count() File /databricks/spark/python/pyspark/instrumentation_utils.py:47, in _wrap_function.<locals>.wrapper(*args, **kwargs) 45 start = time.perf_counter() 46 try: ---> 47 res = func(*args, **kwargs) 48 logger.log_success( 49 module_name, class_name, function_name, time.perf_counter() - start, signature 50 ) 51 return res TypeError: DataFrame.dropDuplicates() got an unexpected keyword argument 'subset'

It works fine if I only pass the list without using it as a keyword argument.

It looks like they changed the function definition to receive a varargs instead of a list but this broke a lot of code for us.

Does somebody else have this problem?

Re: Problem with dropDuplicates in Databricks runtime 15.4LTS

kellys — Fri, 13 Sep 2024 14:08:57 GMT

Wanted to add to this thread. Seeing the same issue. This appears to be recent problem.

Re: Problem with dropDuplicates in Databricks runtime 15.4LTS

RodriGonca — Mon, 16 Sep 2024 14:07:12 GMT

Same thing here, broke a lot of code.

Re: Problem with dropDuplicates in Databricks runtime 15.4LTS

Witold — Mon, 16 Sep 2024 15:40:55 GMT

What happens if you avoid passing it as a named parameter? Like:

data.dropDuplicates(['SOME_COLUMN']).count()

Re: Problem with dropDuplicates in Databricks runtime 15.4LTS

juan_barreto — Mon, 16 Sep 2024 16:30:24 GMT

Hi, As I said, doing that works. But it broke a really big codebase.

Databricks should not be changing the public API of a function in a "stable" release.

Re: Problem with dropDuplicates in Databricks runtime 15.4LTS

szymon_dybczak — Mon, 16 Sep 2024 16:42:51 GMT

Exactly what @juan_barreto said. The public api should be a contract that we can trust and it shouldn't be changed lightly. Imagine codebase with hundreds of notebooks and a developer team agreed to follow convention to use keyword arguments in that particular function. Now you have a problem. Solution is simple, but you need to rewrite your whole codebase.

Re: Problem with dropDuplicates in Databricks runtime 15.4LTS

Witold — Tue, 17 Sep 2024 06:00:38 GMT

Unless is was communicated as a breaking changes between major updates, it would be OK. But I can't find anything in the release notes, so it's a bug.