Walter_C
Databricks Employee
Databricks Employee

The behavior you're observing is likely due to a combination of factors related to how Python executes code and how time is measured. Let's break down the issues and provide some recommendations for more accurate timing:

  1. Resolution of time.time():
    The resolution of time.time() is typically around 1 microsecond on most systems. For very fast operations, this might not be accurate enough.
  2. Overhead of function calls:
    Calling time.time() itself takes some time, which can be significant for very fast operations.
  3. Python interpreter overhead:
    The Python interpreter introduces some overhead, especially when executing small pieces of code repeatedly.
  4. System-level scheduling:
    The operating system may introduce delays between iterations of your loop, leading to inconsistent measurements.
  5. JIT compilation (if using PyPy):
    If you're using PyPy, just-in-time compilation can cause timing variations.


For this you can try:

  1. Use timeit module:
    The timeit module is designed for benchmarking small code snippets and handles many of the issues mentioned above.
  2. Use time.perf_counter():
    For more precise timing, use time.perf_counter() instead of time.time(). It provides higher resolution and is monotonic.

View solution in original post