Methodology

The components of the evaluation form were drawn directly from the language of the Open Government Directive. The evaluation is not a compliance audit, nor does it measure whether or not an agency is implementing open government plans.

The scoring system is intended to evaluate the strength of the overall program in meeting the requirements of the Directive. The requirements are judged on a “0” to “2” scale. If an agency omits a requirement from the plan, it is scored as a “0;” if it includes an aspirational reference to a requirement (a plan to plan to fulfill the requirement), it is scored as a “1;” if the agency fulfills the requirement, it is scored as a “2.” Additionally, if an agency exceeds any individual component, it can be awarded a bonus point for a total of 3 points.

The “basic score” for an agency is either 58 or 60 – depending on whether an agency has original classification authority.  All components of the government that have such authority under President Obama’s Executive Order on Classified National Security Information are assumed to have responsibility for meeting the Directive’s declassification requirements (basic score of 60). If an agency does not have original classification authority, the component is assumed to not apply to that agency and it is not included on the agency’s evaluation form (basic score of 58).

Final rankings are based on the overall score the plan earned (including bonus points) out of the agency’s basic score.

OpenTheGovernment.org recruited and organized volunteers with experience working with agencies and evaluating information policies to evaluate agencies plans from non-profits, academia, and other organizations that serve the public interest. Evaluators were asked to complete their evaluations on worksheets that include space for score justifications. To create a degree of consistency among the evaluations, the evaluators met prior to beginning their work to discuss standards for grading each plan. Additionally, the evaluators used a Google Group to circulate and discuss particularly thorny questions about how an agency should be graded on a particular component.

For example, as the Directive is silent on what constitutes a “significant FOIA backlog,” the evaluators decided to set the threshold as “a backlog at the end of Fiscal Year (FY) 2009 that is larger than 10% of the number of FOIA requests received in FY 2009, as reported in the agency’s annual FOIA report.” The scoring guide at the top of the evaluation form for each agency noted whether or not the agency exceeded this threshold, and directed evaluators to score only the corresponding element (either agency has a plan to reduce the backlog, or agency has no significant backlog).

After the worksheets were completed, OpenTheGovernment.org checked each form to make sure all of the components were scored as directed in the scoring guide (available at the top of each agency's evaluation). If the score on any particular component seemed out of line with how other evaluators were scoring similar plans, OpenTheGovernment.org alerted the evaluator to the discrepancy, and asked for further justification. The overview observations reported on this site were pulled together by OpenTheGovernment.org from the evaluators, and discussed with and refined by the evaluators.

Comments