DLT Event Logs

[Djelany]

Hi,

Does anyone know what details:planning_information:technique_information[0]:cost under planning_information event type means in my DLT workflow system event logs? For context, I'm trying to track the cost per run of my DLT workflow and I do not have Unity Catalog enabled. 

Thank you!

[mtajmouati · Accepted solution]

Hi ! 

The cost field under planning_information in your DLT (Delta Live Tables) workflow system event logs represents an estimated cost for the particular maintenance type operation. In this context, MAINTENANCE_TYPE_COMPLETE_RECOMPUTE indicates that a complete recompute operation is planned, and the cost value (48.0) likely signifies the estimated computational cost for this operation.

Explanation of Fields:

maintenance_type: Type of maintenance operation being performed.

is_chosen: Indicates if this technique was chosen for execution.

is_applicable: Indicates if this technique is applicable to the current scenario.

cost: Estimated computational cost for this maintenance operation.

table_id: Identifier for the target table on which this maintenance operation is performed.

[Djelany]

Thank you! A quick follow up question, is there a way to track the full cost of the run by querying the system event logs instead of just the maintenance cost? Or is that only possible by querying the billable usage path in S3 (in my case)?

[adriennn]

you can enable system.billing schema and see the costs of the runs from the usage table.