topic Re: Strange object returned using sparklyr in Data Engineering

Strange object returned using sparklyr

Jeff1 — Wed, 23 Mar 2022 15:39:55 GMT

Community

I'm running a sparklyr "group_by" function and the function returns the following info:

# group by event_type

acled_grp_tbl <- acled_tbl %>%

group_by("event_type") %>%

summary(count = n())

Length Class Mode

src 1 src_spark list

ops 4 op_group_by list

sdf_cache_state 2 -none- environment

spark_dataframe 1 -none- function

schema_cache_state 2 -none- environment

schema 1 -none- function

Not what I expected and have no idea what it is telling me. Other than my group_by isn't working. Any ideas.

Jeff

Re: Strange object returned using sparklyr

Hubert-Dudek — Wed, 23 Mar 2022 21:33:41 GMT

I think "event_type" should be without quotation marks

Re: Strange object returned using sparklyr

Jeff1 — Thu, 24 Mar 2022 12:21:44 GMT

I should have deleted the post. While your are correct "event_type" should be without quotes the problem was the Summary function. I was using the wrong function it should have been "summarize."

Re: Strange object returned using sparklyr

Hubert-Dudek — Thu, 24 Mar 2022 14:15:15 GMT

yes I haven't spotted that as names are similar 🙂