topic Re: Table Refresh UI Error in Data Engineering

Table Refresh UI Error

BenLambert — Wed, 17 May 2023 11:12:03 GMT

Within the UI it is possible to "Select tables for refresh" for a specific Delta Live Tables Workflow. I often use it to make a full refresh on smaller tables during development. Unfortunately, when an error occurs during the full refresh on selected tables, the pipeline is not displayed within the UI anymore and I can't retry the full refresh on specific tables. In that case it is only possible to full refresh the whole pipeline. Is this a known issue/is there a planned fix?

Re: Table Refresh UI Error

jose_gonzalez — Tue, 06 Jun 2023 18:37:41 GMT

Could you please share the full error stack trace? it will help us to narrow down the issue

Re: Table Refresh UI Error

BenLambert — Wed, 14 Jun 2023 13:23:02 GMT

In this case we just need to cause an error of any type where the pipeline doesn't execute and then in the UI it no longer shows an option to refresh a specific table, leaving only a full refresh as an option. In this case we removed the specification of which instance type the pipeline uses and this causes a quota error and yields the situation that I was describing.

Here is a screenshot that shows when the pipeline fails that the UI no longer offers a way to select which table to refresh.