topic Re: GroupBy in delta live tables fails with error "RuntimeError: Query function must return either a Spark or Koalas DataFrame" in Data Engineering

GroupBy in delta live tables fails with error "RuntimeError: Query function must return either a Spark or Koalas DataFrame"

PrebenOlsen — Fri, 26 Aug 2022 11:31:47 GMT

I have a delta live table that I'm trying to run GroupBy on, but getting an error: "RuntimeError: Query function must return either a Spark or Koalas DataFrame".

Here is my code:

@dlt.table
def groups_hierarchy():
 
     df = dlt.read_stream("groups_hierarchy_vw")
     return(df
        .select("id","name",split("path","/").alias("groups_in_path"),posexplode(split("path","/")).alias("pos","value"))
        .drop("val")
        .select("id","name",concat(lit("group"),"pos").alias("group_name"),expr("groups_in_path[pos]").alias("val"))
        .groupBy([df.id, df.name])

Edit:

Something as simple as the following works just fine (you will notice I I am now reading a regular table and not a stream, just for testing purposes):

@dlt.table
def groups_hierarchy():
     return dlt.read("streaming_silver").groupBy("id").count()

And it works fine when I apply my select statements and transformations, but the absolute last .groupBy() seems to convert it to a non Spark/Koalas DataFrame

@dlt.table
def groups_hierarchy():
     return dlt.read("streaming_silver").select("id","name",split("path","/").alias("groups_in_path"),posexplode(split("path","/")).alias("pos","value")).drop("val").select("id","name",concat(lit("group"),"pos").alias("group_name"),expr("groups_in_path[pos]").alias("val")).groupBy("id")

Re: GroupBy in delta live tables fails with error "RuntimeError: Query function must return either a Spark or Koalas DataFrame"

PrebenOlsen — Mon, 29 Aug 2022 06:58:27 GMT

No, there is no join in my code above. The problem occurs at the very last line, when trying to use a .groupBy.

This works fine when reading a non-streaming view, but fails as soon as it is a stream. Is groupBy not supported yet for streams?

Re: GroupBy in delta live tables fails with error "RuntimeError: Query function must return either a Spark or Koalas DataFrame"

Vidula — Thu, 15 Sep 2022 10:33:44 GMT

Hi @Preben Olsen

Does @Debayan Mukherjee response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?

We'd love to hear from you.

Thanks!

Re: GroupBy in delta live tables fails with error "RuntimeError: Query function must return either a Spark or Koalas DataFrame"

Debayan — Fri, 26 Aug 2022 14:56:00 GMT

Are you trying to join two live tables using python?

Please refer the code here: https://stackoverflow.com/questions/73112299/is-there-a-way-to-join-two-live-tables-on-delta-live-tables-using-python

Please let us know if this helps or if any further clarification is needed on the same.

Re: GroupBy in delta live tables fails with error "RuntimeError: Query function must return either a Spark or Koalas DataFrame"

Debayan — Fri, 02 Sep 2022 21:19:07 GMT

Could you please try to update your spark version. For few of the used cases , spark version 2.3.0 was throwing the same error which was fixed in 2.4.0.

Please refer:

https://issues.apache.org/jira/browse/SPARK-24156