mark_ott
Databricks Employee
Databricks Employee

The reason your profile or drift table shows a window starting earlier than your actual data date (February 24 instead of March 1) is due to how granularity and time-window alignment work in your monitoring setup.

Why the window starts at February 24

When you set a weekly granularity, the system automatically groups data by calendar week boundaries, not by your dataset’s start date. In most monitoring systems like Azure ML or Visier, weekly granularity typically follows the ISO week convention (Monday through Sunday).

So when your data begins on March 1, 2025 (Saturday):

  • The week that includes March 1 actually starts on Monday, February 24, 2025, and ends on Sunday, March 2, 2025.

  • Therefore, your first monitoring window is labeled as starting on 2025-02-24T00:00:00Z and ending on 2025-03-03T00:00:00Z, even if no data exists for February 24–28.​

This ensures consistent week-to-week comparisons across your system, helping drift detection tools aggregate data uniformly within predefined temporal buckets.

How granularity affects your analysis

Granularity determines how the data is grouped and summarized for drift analysis or profiling :​

  • Daily granularity evaluates drift per day — detailed but can be noisy.

  • Weekly granularity smooths short-term variance, giving cleaner insight into slow changes or long-term drift.

  • Monthly granularity aggregates heavily — useful for stable, long-term tracking.

Setting a coarser granularity (like weekly) enlarges each time window, which may cause start times to appear earlier than your visible data because of anchor alignment to natural calendar intervals.

Best practices for interpreting dashboard insights

To make the best use of your drift/profiling dashboard:

  1. Confirm window alignment: Understand that windows cover fixed periods (weeks, months) defined by system rules, not your dataset boundaries.

  2. Filter empty windows: Ignore windows with no actual data records loaded; drift metrics will naturally be zero or incomplete.

  3. Use weekly or daily granularity depending on data volume:

    • For continuous, high-frequency model input — daily granularity gives precision.

    • For moderate datasets (like yours) — weekly granularity balances stability and detectability.

  4. Leverage feature-level drift metrics: Focus on which input features are showing the largest change in distribution between windows; these usually explain model degradation or instability.​

In short, the February 24 start time appears because your chosen one-week granularity anchors to full calendar weeks rather than your actual data start date. This is normal and ensures consistent comparison across all future monitoring windows.