Databricks Community

leungi

BackgroundUsing R language's {sparklyr} package to fetch data from tables in Unity Catalog, and faced the error below.Tried the following, to no avail:Using memory optimized cluster - e.g., E4d.Using bigger (RAM) cluster - e.g., E8d.Enable auto-scali...

leungi

End goal is to apply OPTIMIZE and ZORDER table.However, one of the columns to be ZORDER doesn't have stats collected.Running ANALYZE generates the error below.QueryANALYZE TABLE <catalog>.<schema>.<table> COMPUTE STATISTICS FOR COLUMNS my_col_1, my_c...

leungi · 06-13-2024

An init script is used to install system libraries, per below.Adding the script to a Personal Compute consistently works. The same script is added to a Workflows job via cluster config, which intermittently fails, as shown in error message below.Both...

leungi · 05-10-2024

The `Library` option in cluster config allows installation of language-specific libraries - e.g., PyPi for Python, CRAN for R.Some of these libraries - e.g., `sf` - require system libraries - e.g., `libudunits2-dev`, `libgdal-dev`.How may one install...

leungi · 05-10-2024

The following doc suggests the ability to add column comments during MV creation via the `column list` parameter.Thus, the SQL code below is expected to generate a table where the columns `col_1` and `col_2` are commented; however, this is not the ca...

leungi

@Kaniz_Fatma , thanks for the detailed suggestions.I believe the first reference relates to the issue; however, after adjusting spark.driver.maxResultSize to various values - e.g., 10g, 20g, 30g - a new error ensues (see below).The operation involv...

leungi

@Kaniz_Fatma , thanks for the detailed suggestions.1. Running SHOW COLUMN STATS yields the following error:[PARSE_SYNTAX_ERROR] Syntax error at or near 'CHECK'. SQLSTATE: 42601 line 1, pos 0== SQL ==CHECK COLUMN STATS <catalog>.<schema>.<table> ^^^Ra...

leungi

Thanks for the suggestion @amr.Courtesy of a DBX solution engineer, the key was to remove all the files in the /var/lib/apt/lists/ directory to force apt to download fresh package lists during subsequent update.Init Script#!/bin/bashset -euxo pipefai...

leungi · 05-31-2024

Managed to get it working per your guidance For future me:1. Create a SQL notebook. CREATE OR REFRESH LIVE TABLE my_dlt_mv (col_1 STRING COMMENT 'column_1', col_2 STRING COMMENT 'column_2') COMMENT 'MV of a streaming table.' AS ( SELECT col_1 , c...

leungi · 05-31-2024

Noted.I tried the following in both SQL Editor and Notebook and got the same outcome: %sql USE CATALOG my_catalog; USE SCHEMA my_schema; DROP MATERIALIZED VIEW IF EXISTS my_mv; CREATE MATERIALIZED VIEW my_mv (col_1 STRING COMMENT 'Column 1', col...

Databricks Community

User Stats

User Activity

Spark Out of Memory Error

Error on ANALYZE TABLE [...] COMPUTE STATISTICS FOR COLUMNS [...]

Init Script Fails Intermittently on Workflow Job

Install system libraries on the cluster

Unable to add column comment in Materialized View (MV)

Re: Spark Out of Memory Error

Re: Error on ANALYZE TABLE [...] COMPUTE STATISTICS FOR COLUMNS [...]

Re: Init Script Fails Intermittently on Workflow Job

Re: Unable to add column comment in Materialized View (MV)

Re: Unable to add column comment in Materialized View (MV)