Facing Data Truncation Issues in Databricks Dashboards

Akshay_Petkar — Fri, 31 May 2024 10:44:51 GMT

I'm encountering data truncation in my Databricks dashboards. I'm working with a large dataset, and the dashboard only displays a limited number of (truncated) rows.
let's take a dataset containing 1 million sales records. The dashboard currently only displays a limited number of rows, which affects the accuracy of important metrics such as total sales. Is there a way to configure the dashboards to handle larger datasets and ensure that they display the complete picture, including all 1 million records in visualizations?

Re: Facing Data Truncation Issues in Databricks Dashboards

NandiniN — Sat, 01 Jun 2024 11:51:23 GMT

Hi @Akshay_Petkar ,

This is simple in Databricks SQL, just uncheck LIMIT 1000 in the drop down.

https://docs.databricks.com/en/sql/get-started/visualize-data-tutorial.html

Limit 1000 is selected by default for all queries to ensure that the query returns at most 1000 rows. If a query is saved with the Limit 1000 setting, this setting applies to all executions of the query (including within dashboards). If you want to return all rows for this query, you can unselect LIMIT 1000 by clicking the Run (1000) drop-down. If you want to specify a different limit on the number of rows, you can add a LIMIT clause in your query with a value of your choice.

The query result displays in the Results tab.

Same way in notebook you can click on download full results.

But it won't preview the entire data on the UI.

topic Re: Facing Data Truncation Issues in Databricks Dashboards in Warehousing & Analytics

Facing Data Truncation Issues in Databricks Dashboards

Re: Facing Data Truncation Issues in Databricks Dashboards