topic Re: DLT Event Logs in Data Engineering

DLT Event Logs

Djelany — Fri, 05 Jul 2024 21:11:43 GMT

Hi,

Does anyone know what details:planning_information:technique_information[0]:cost under planning_information event type means in my DLT workflow system event logs? For context, I'm trying to track the cost per run of my DLT workflow and I do not have Unity Catalog enabled.

Thank you!

Re: DLT Event Logs

mtajmouati — Fri, 05 Jul 2024 23:36:57 GMT

Hi !

The cost field under planning_information in your DLT (Delta Live Tables) workflow system event logs represents an estimated cost for the particular maintenance type operation. In this context, MAINTENANCE_TYPE_COMPLETE_RECOMPUTE indicates that a complete recompute operation is planned, and the cost value (48.0) likely signifies the estimated computational cost for this operation.

Explanation of Fields:

maintenance_type: Type of maintenance operation being performed.

is_chosen: Indicates if this technique was chosen for execution.

is_applicable: Indicates if this technique is applicable to the current scenario.

cost: Estimated computational cost for this maintenance operation.

table_id: Identifier for the target table on which this maintenance operation is performed.

Re: DLT Event Logs

Djelany — Mon, 08 Jul 2024 13:00:01 GMT

Thank you! A quick follow up question, is there a way to track the full cost of the run by querying the system event logs instead of just the maintenance cost? Or is that only possible by querying the billable usage path in S3 (in my case)?

Re: DLT Event Logs

adriennn — Fri, 06 Sep 2024 07:43:36 GMT

you can enable system.billing schema and see the costs of the runs from the usage table.