Guideline 7.3-- Defining Acceptance Criteria. The quality assurance (QA) and
validation criteria applied to the RBP system should be documented.
Discussion. The QA and validation criteria applied to an RBP effort should be
commensurate with the scale of the effort. The general goal is to ensure that the RBP
system has been properly applied and yields results that meet the original purposes
defined by the user. The QA and validation checks should verify that (1) assumptions
have been documented, (2) the significance of assumptions and values of parameters
used in the RBP system have been adequately examined and described, (3) there are
no significant signs of bias, and (4) any interdependencies among decision options
have been defined and properly handled.
Guideline 7.4-- Assuring Quality. Documentation of the prioritization model, input data
(work packages), and the preliminary RBP prioritization results should be subject to
QA and other validation checks before further use of the preliminary results.
Deficiencies should be corrected prior to proceeding with the remainder of the RBP
Discussion. The QA and validation checks provide confidence that the prioritization
model was properly implemented and that the model produced results that pass tests
for reasonableness before the results are compiled for presentation to the decision
makers who requested the RBP. If scores were generated by a " coring team,"some
subset of that team can be used as the quality assurance team provided that
individuals do not review their own work. To counter and minimize " aming,"peer
review or other such techniques should be pursued. Peer review can ensure that the
highest professional standards are maintained. Therefore, procedures should be
developed to maximize the use of quality assurance and validation checks.
Guideline 7.5-- Documentation. The report produced at the conclusion of an RBP
effort should thoroughly define the original user requirements (purpose statement and
decision options), the RBP process applied, the input data used, and the results
derived. The results should be presented in a form that meets the original
requirements specified by the user.
Discussion. There are two key aspects to documentation. One is to provide the
necessary documentation to allow the decision maker to have confidence in using the
results as a basis for decisions. The second is to defend the results to a broad
audience of peer reviewers and interested parties.
a. Documentation to Support Decision Makers. An RBP system is useful only to the
extent that its outputs are relevant in the context of an actual decision-making
effort. A well-designed RBP process allows a wide range of potentially useful
outputs to be generated for decision makers. Specific user needs are defined at
the start of the RBP process and are captured in the decision options defined by
the end user. The RBP results should be selected and presented to maximally
achieve the purposes for which the system was developed. Thus, the purpose