How to display Sparklyr table in a clean readable format similar to the output of display()?

SG_ — Thu, 17 Mar 2022 14:40:11 GMT

There exist a Databricks’s built-in display() function (see documentation here) which allow users to display R or SparkR dataframe in a clean and human readable manner where user can scroll to see all the columns and perform sorting on the columns. See image below:

Is there a way to display tables that are output from sparklyr in such a manner in Databricks? Obviously I am not able to use function directly on the sparklyr table hence I attempted to convert the sparklyr table into a R dataframe before using display() function but was hit with java.lang.IllegalArgumentException: Invalid type N error.

Re: How to display Sparklyr table in a clean readable format similar to the output of display()?

rich_goldberg — Tue, 21 Feb 2023 19:47:52 GMT

I found that the display() function returned this issue when it came across date-type fields that were NULL. The following function seemed to fix the problem:

library(tidyverse)
library(lubridate)
 
display_fixed = function(df) {
    df %>% 
        mutate( across( where(is.instant), ~if_else( is.na(.x), as_datetime("1900-01-01"), as_datetime(.x)) ) ) %>% 
        display()
}

Try the following in your test example:

dat2 %>% collect(n=10) %>% display_fixed()

topic How to display Sparklyr table in a clean readable format similar to the output of display()? in Data Engineering

How to display Sparklyr table in a clean readable format similar to the output of display()?

Re: How to display Sparklyr table in a clean readable format similar to the output of display()?