topic Re: Databricks Job ID's increasing in massive sequence gaps in Data Engineering

Databricks Job ID's increasing in massive sequence gaps

JoeWMP — Tue, 22 Feb 2022 20:34:51 GMT

Has anyone seen something like this before? Today around midnight, our Job ID's started increasing in increments of quadrillions - was this a new change to how Job ID's are generated?

Re: Databricks Job ID's increasing in massive sequence gaps

JoeWMP — Tue, 22 Feb 2022 20:38:25 GMT

If it helps, we've seen this in multiple Azure Databricks environments

Re: Databricks Job ID's increasing in massive sequence gaps

-werners- — Wed, 23 Feb 2022 08:23:47 GMT

I guess that is possible, maybe databricks rolled out an update or changed the ID logic.

Re: Databricks Job ID's increasing in massive sequence gaps

JoeWMP — Wed, 23 Feb 2022 15:51:19 GMT

Yeah, it can go up to int64 so they aren't invalid values, but it's very strange that it would increase in increments of 1 and now is jumping sequences in large increments. In an account that we had daily processing for a about a year, job id value only made it to around 4 million - now we're in the quadrillions or so

Re: Databricks Job ID's increasing in massive sequence gaps

Ravi — Wed, 23 Feb 2022 18:13:48 GMT

@Joe Swanson @Werner Stinckens Yes, we have started rolling out the job ID logic as mentioned here to all workspaces. https://docs.databricks.com/release-notes/product/2021/august.html#jobs-service-stability-and-scalability-improvements

Re: Databricks Job ID's increasing in massive sequence gaps

JoeWMP — Thu, 24 Feb 2022 17:26:51 GMT

Thank you Ravi! Glad that this confirms my understanding