Hello,
I have developed a dashboard for monitoring compute costs using system tables, allowing tracking of expenses by Cluster Name (user created name), Job Name, or Warehouse Name. However, with the introduction of the new shared serverless compute, particularly with the SKU name "ENTERPRISE_JOBS_SERVERLESS_COMPUTE_US_EAST_N_VIRGINIA," four billing origin products are being generated.
The first product is 'SQL,' which seems to pertain to Delta Live Tables (DLT), as the only data associated with it is a deltapipeline_id. It is unclear why this is categorized under 'SQL.' The second product is 'JOBS,' which includes a job_id. However, there is no job corresponding to this job_id in the data, nor is there a DLT pipeline associated with the pipeline_id.
Could someone provide guidance on how to accurately track and categorize these costs?
Thank you.