topic Re: Can I change from default showing first 1,000 to return all records when query? in Data Engineering

Can I change from default showing first 1,000 to return all records when query?

Thanapat_S — Wed, 16 Nov 2022 05:54:26 GMT

I have to query a data for showing in my dashboard.

But it truncated the results and showing only first 1,000 rows.

In the dashboard view, there is no option to re-execute with maximum result limits.

I don't want to switch back to standard view and click to re-execute everytime.

So, I would like to know if I can change default from showing only 1,000 records to return all records from query.

More information

Workspace: Data Science & Engineering

Language: Python (PySpark)

Databricks Runtime: 9.1 LTS (includes Apache Spark 3.1.2, Scala 2.12)

Re: Can I change from default showing first 1,000 to return all records when query?

Srihasa_Akepati — Wed, 16 Nov 2022 19:38:33 GMT

@Thanapat Sontayasara 10000 limit is available as an option in the notebook(which propagates to dashboard after its run in notebook) while 1000 rows still is the default. 10000 limit is experimental and it can be made as default depending on the number of use cases from customers which may take some time.

Re: Can I change from default showing first 1,000 to return all records when query?

Thanapat_S — Sat, 19 Nov 2022 09:10:52 GMT

Hi! @Kaniz Fatma Sorry for late reply.

Thank you for your answer.

But the problem is that my query is running in Databricks Data Science & Engineering.

I understand that your reference is based on Databricks SQL workspace.

Re: Can I change from default showing first 1,000 to return all records when query?

Thanapat_S — Sat, 19 Nov 2022 09:13:28 GMT

Hi! @Srihasa Reddy Akepati Sorry for late reply.

Interesting! Can you share with us about how to change or set the 10,000 as default when running command like display(df) and return 10,000.

With screenshot or document guideline shared would be really appreciated!

Re: Can I change from default showing first 1,000 to return all records when query?

Thanapat_S — Sat, 19 Nov 2022 09:15:38 GMT

Hi! @Kaniz Fatma Sorry for late reply.

I am still don't find any solution to change the default query limit when using display(df).

If I got the solution, I will share with the community ASAP.

Thank you for your support 🙂

Re: Can I change from default showing first 1,000 to return all records when query?

Thanapat_S — Sat, 19 Nov 2022 10:18:01 GMT

Hi! @Kaniz Fatma.

Yes, definitely the same question as mine.

But, still, I don't find any guideline of how to change the default like mention in that thread.

Do you have any clue about "the changes need to occur at the shard level."?

I have no idea what is the shard level. Or you have any reference document for me to work on.

Re: Can I change from default showing first 1,000 to return all records when query?

Srihasa_Akepati — Tue, 22 Nov 2022 05:19:30 GMT

@Thanapat Sontayasara Default is still 1000. Let me check internally if there is a way to change the default.

Re: Can I change from default showing first 1,000 to return all records when query?

Srihasa_Akepati — Tue, 22 Nov 2022 15:47:41 GMT

@Thanapat Sontayasara 1000 is still the default. Default of 10k rows will be made available by Jan.

Re: Can I change from default showing first 1,000 to return all records when query?

Thanapat_S — Tue, 22 Nov 2022 16:03:58 GMT

Hi @Srihasa Reddy Akepati

That's a useful information and also a good news.

I'm looking forward to that feature release.

Thank you so much for your support. 😄

Re: Can I change from default showing first 1,000 to return all records when query?

jngnyc — Tue, 03 Jun 2025 15:25:59 GMT

I found this explanation helpful: