Hello all,
I'd like to report what appears to be an incorrectly keyed question in the assessment for the Evaluating AI Agents course.
Question:
"According to the lecture on evaluating AI agents, which of the following is described as a key reason why evaluation must be treated as a continuous process rather than a one-time validation step?"
My answer (marked incorrect, score 0):
"Production agents encounter diverse user queries, usage patterns change over time, and new failure modes emerge that were not anticipated during development."
The answer the platform marks correct:
"Unity Catalog enforces version limits on registered models, requiring re-evaluation before each new version is promoted."
I believe the keyed answer is incorrect for two reasons.
First, Unity Catalog does not enforce a version limit on registered models that triggers mandatory re-evaluation. Even if such a limit existed, a registry constraint would be an operational detail rather than a reason evaluation must be continuous.
Second, the keyed answer contradicts other questions within the same assessment. The question on offline versus online evaluation is keyed to: "Offline evaluation datasets may not fully represent real user behavior, can become stale as usage patterns evolve, and cannot capture issues that only emerge at scale." The question on the offline/online feedback loop is keyed to a sequence built on analysing production traces for failures and edge cases not anticipated during development. Both reflect the same reasoning as the answer marked incorrect here.
I've attached a screenshot showing the question, my selection, and the feedback.
Could someone from the course team confirm whether this key needs updating? And has anyone else encountered the same result on this question?
