How to display Sparklyr table in a clean readable format similar to the output of display()?

[SG_]

There exist a Databricks’s built-in display() function (see documentation here) which allow users to display R or SparkR dataframe in a clean and human readable manner where user can scroll to see all the columns and perform sorting on the columns. See image below:

Is there a way to display tables that are output from sparklyr in such a manner in Databricks? Obviously I am not able to use function directly on the sparklyr table hence I attempted to convert the sparklyr table into a R dataframe before using display() function but was hit with java.lang.IllegalArgumentException: Invalid type N error.

 

[rich_goldberg]

I found that the display() function returned this issue when it came across date-type fields that were NULL. The following function seemed to fix the problem:

library(tidyverse)
library(lubridate)
 
display_fixed = function(df) {
    df %>% 
        mutate( across( where(is.instant), ~if_else( is.na(.x), as_datetime("1900-01-01"), as_datetime(.x)) ) ) %>% 
        display()
}

Try the following in your test example:

dat2 %>% collect(n=10) %>% display_fixed()