Acceptance and Evaluation Criteria

Acceptance criteria and evaluation criteria can almost be considered as two separate considerations, but they are presented together in the BABOK and I can see why. The BABOK states that acceptance criteria are the basis for determining whether a solution is acceptable to stakeholders based on identified requirements, and the evaluation criteria are about how multiple possible solutions are compared to pick the best one. So both are about solutions, but one (acceptance criteria) is about meeting requirements and the other (evaluation criteria) is potentially exceeding them (if one meets requirements by “more” than another).

Another way to think about evaluation requirements is to note that defined requirements of value may not exist. That is, an organization may identify requirements and develop solutions that meet those requirements, but the organization may or may not have, say, a specific cost a project needs to stay under or a specific benefit a solution needs to realize. Instead, in such a situation, the costs and benefits are compared and an entrepreneurial judgment is made about whether or not to proceed.

Still another complication is when potential solution approaches are sufficiently different that the participants need to create some kind of hybrid scoring system that allows comparison of different mixes of features that cannot be compared directly. On the one hand, many criteria can be translated into monetary terms (abstractly, at least), but on the other hand it doesn’t always occur to people to do so, and attempts at it might not be all that accurate.

Functional requirements are more often amenable to discrete evaluation with yes/no, go/no-go -type answers than are non-functional requirements. For example, you might specify that “a process must run to completion within 2.5 seconds,” but how do you judge whether a solution or element is more or less elegant or modular? Some non-functional requirements can be objectively evaluated, as a measure of robustness might be expressed as “the system must achieve 95% operational uptime,” but it may be that being able to express a requirement in that way inherently transforms it into a functional requirement. A review of the non-functional requirements listed here shows that some can be directly evaluated and some cannot readily be, so I will leave this idea as something to reflect upon.

My framework emphasizes iteration and communication within and between phases. Communication and iteration within phases, especially in the first three, involves having analysts learn from customers, document their findings, submit them to the customers for review, and modifying the documented findings until the customers agree that the analysts have everything right. This means that a form of acceptance is embedded into each phase when customers accept the definition(s) of the intended use, the findings of discovery and data collection activities, and the expression of requirements. After that the emphasis shifts to evaluation criteria.

The requirements traceability matrix links all elements of the process or product under investigation to elements in the previous and subsequent phases in both directions. This ensures that all work is targeted to items that address the intended use, that all requirements are addressed in the design, that all design elements are implemented, and that all implementations are tested. The linking encompasses acceptance criteria in a way and is a form of validation (whether the developed solution addresses the identified problem or use), but the requirements is really where the acceptance criteria are explicitly baked in.

A good requirement will usually be written in a way that includes how it will be judged to have been met. After that there are endless ways of testing all elements of the solution, and these most often (but not always) involve forms of verification (whether the developed solution functions as intended). Certain kinds of implementation may involve numerous verification tests (like automated unit tests for software code) that aren’t written by the analysts and customers, but that will vary with the situation.

If this article seems to wander through a bunch of different concepts in no particular order, let me return to the original definition from the BABOK. Acceptance criteria drive yes/no decisions that can often but not always be automated. Evaluation criteria allow comparison of potential solutions either directly or indirectly.

This entry was posted in Tools and methods and tagged , , , , . Bookmark the permalink.

Leave a Reply