AbhayPSingh
Databricks Employee
Databricks Employee

More or less repeating what Mark said and adding some additional thoughts.

Why the Window Starts from February 24

The reason you're seeing a window starting from February 24 (even though your data starts March 1) is because monitoring systems align time windows to standard calendar boundaries rather than your data's actual start date.

  When you set a 1-week granularity:

  - The system creates windows aligned to calendar weeks (typically Monday-Sunday or Sunday-Saturday)

  - March 1, 2025 is a Saturday

  - The calendar week containing March 1 actually starts on February 24 (Monday) and ends on March 2 (Sunday)

  - This is why your first window shows: February 24 00:00:00 to March 3 00:00:00

 

  How Granularity Works

 

  Granularity in monitoring systems determines:

 

  1. Window Size: The time period for aggregating metrics

    - 1 week = 7-day windows

    - Windows are fixed to calendar boundaries

  2. Alignment: Windows snap to standard intervals

    - Weekly: Aligns to start of week (Monday or Sunday)

    - Daily: Aligns to midnight

    - Monthly: Aligns to first of month

  3. Coverage: Each window includes all data points within that time range

    - Your March 1-2 data falls into the Feb 24 - Mar 2 window

    - March 3-9 data goes into the next window

    - March 10-14 data goes into a third window

 

  Best Practices for Granularity Selection

 

  Choose granularity based on:

 

  1. Data Volume

    - High volume (1000s/day): Use daily or weekly

    - Medium volume (100s/day): Use weekly or bi-weekly

    - Low volume (<100/day): Use monthly

  2. Change Detection Needs

    - Rapid drift detection: Use daily

    - Stable patterns: Use weekly/monthly

    - Seasonal patterns: Match the seasonality period

  3. Business Requirements

    - Real-time monitoring: Daily or shorter

    - Trend analysis: Weekly/monthly

    - Reporting cycles: Align with business reporting

 

  Getting Insights from the Dashboard

 

  To maximize dashboard value:

 

  1. Focus on Drift Metrics

    - Look for sudden spikes in drift scores

    - Compare consecutive windows for trends

    - Set alerts for significant drift thresholds

  2. Analyze Feature Statistics

    - Monitor mean/median shifts

    - Check distribution changes (histograms)

    - Track null rates and data quality

  3. Time-based Patterns

    - Compare weekday vs weekend patterns

    - Identify seasonal trends

    - Look for gradual vs sudden changes

  4. Actionable Insights

    - Prioritize features with highest drift

    - Correlate drift with model performance

    - Document when/why drift occurs

 

  Recommendations for Your Setup

 

  Given your March 1-14 data with weekly granularity:

 

  1. Expect 3 windows:

    - Feb 24 - Mar 2 (contains Mar 1-2 data)

    - Mar 3 - Mar 9 (full week of data)

    - Mar 10 - Mar 16 (contains Mar 10-14 data)

  2. Consider adjusting granularity if:

    - You need faster drift detection → Use daily

    - You have limited data → Use bi-weekly

    - You want smoother trends → Use monthly

  3. Handle partial windows:

    - First/last windows may have less data

    - Consider minimum data thresholds for reliable metrics

    - Document expected vs actual data coverage

 

  The February start date in your window is completely normal behavior - it's the system ensuring consistent, comparable time  windows aligned to calendar boundaries rather than your data boundaries.