System.billing.usage table - cannot match job_id from databricks api/UI

[rocky5]

Hello, I have multiple continuous jobs that are running for many days (Kafka stream), however querying System.billing.usage table by job_id from UI or databricks job api not return any results for those jobs.

1. What is the reason behind that?

2. If I will overwrite job with new version, job_id will change?

3. If yes, how can I track job_id changes to get the entire dbu consumption per specific job name?

BR

Piotr

[Walter_C]

• The System.billing.usage table in Databricks is updated every hour, and it only includes completed tasks. If your jobs are long-running and have not completed yet, they will not appear in this table. Additionally, the table only retains data for the past 30 days, so if your jobs have been running for longer than that, their usage data may not be available.

• Yes, if you overwrite a job with a new version, the job_id will change. The job_id is unique to each job run, so any changes to the job will result in a new job_id.

[rocky5]

I have to disagree. I have continued my job which was failing a few times a day (connectivity issues). So it should have an entry in usage table. Maybe it has, but for sure none of job_id from UI/API does not match its name. job_id shouldn't change every job run, its run_id which should change. I cannot accept your answer.

[rocky5]

Databricks, any update on this?

[Walter_C]

Hello, you are right, apologize for my misunderstanding as you have mentioned the job id persist, is the job run id the one that will be changed. So your issue is that as the job is continuing running the information of that run is not showing in the billing table or the API call? which API are you using?

[rocky5]

Job Id does not exist in billing table. All continues jobs missing