Staff Resources

Process - From Data Definition to Report


Annual reports derive from data collected by each program as defined in the Data Manuals. The report content defined by the Report Guidance documentation developed each year to respond to Board questions that work to support continuous improvement and program accountability. For those reports to inform Board policy and oversight, the data contained in them are codified through work done by the Data Group and subject matter experts. Data Manuals define the data elements to be collected and become the guidebook for program and IT personnel use when designing their data collection system.

Annual Cycle For Creation/Approval of Data Manuals

Steps in Data Manual Development

Steps

Date

What

Who

1 (if proposed DM),

5 (if developed DM)

May Board Meeting*

Action (to move forward or not) on proposed DM

  • Presented by PESB staff
  • Explicitly tied to standards / aligned with projected data needs

Board review / discussion of projected data manuals


Action (to adopt or not) on developed DM

  • As meets criteria of / vetted by Data Group
  • As distributed for input by approved program associations

Public comment on DM development and/or DM being approved


Board

2a

May – March

  • PESB staff and practitioners with expertise relevant to the collection define elements of Board-approved Data Manual in development. 
  • Data Group begins work on how / whether identified elements are measurable and implementable

  • PESB staff
  • Data Group
  • practitioners with relevant expertise

2b

October

General Policy Meeting and continued “framework” dialogue among WACTE / WCEAP / ESA preparation program professional organization(s) and PESB Executive Director to review: 

  • current DM in progress
  • review timeline for future DM development and 
  • collective program / association discussion on own desired collection / elements supplemental to PESB approved / planned collection(s) 

PESB Executive Director in concert with 

  • WACTE
  • WCEAP
  • ESA preparation program professional organization(s)

2c

October - March

If preparation programs collectively reach consensus on new data elements for desired inclusion in the data manual to be approved by the PESB, or development of a supplemental collection / data manual, PESB staff assistance is available for: 

  • convening program / association representatives as a Metrics Group for defining the collection 
  • assisting in element development according to PESB and Data Group requirements 
  • PESB Staff
  • Data Group

3

by April 15

DM(s) final version that has met PESB and Data Group requirements due for inclusion in May Board packet

PESB Staff

4

April 16 - May Board meeting

WACTE / WCEAP / ESA programs review proposed DM with membership in preparation for comments at May Board meeting

  • WACTE
  • WCEAP
  • ESA preparation program professional organization(s)

5 (if developed DM),

1 (if proposed DM)

May Board Meeting

See Step 1 - Action on developed DM

Board

6

June 1

  • Release adopted DM(s) to programs for program data collection
  • Release completed Report Guidance to programs for annual report preparation

PESB Staff

7

September 1

Begin adopted DM(s) collection

Program Staff

8

Varies - see due dates in Report Guidance document

Due dates for reports to be submitted to PESB

PESB Staff

* 2013 will be a sole exception to this timeline, with a proposed new data manual on selection coming forward to the Board in July. In addition to annual approval of new and completed collections in May, unforeseen need for new data elements (e.g. federal or state legislative mandates) could be brought before the Board at another regularly-scheduled Board meeting.


Working Timeline

This timeline follows the development of current and proposed data manuals. These dates are subject to change at Board direction. PESB staff will report to the Board the data submitted through the annual reporting process each March.  


Data Manuals

Purpose

Proposed data manuals are explicit in their connection to Board mandates, which include existing Board expectations (program standards), federal rules, and / or state law as passed by the Legislature. Data Manuals enumerate the data elements that preparation programs collect each year and serve as the foundation for the Data Project

Data Manuals (DM) define statewide data collections, i.e. a data manual commits all PESB-approved educator preparation programs to the collection. Each Data Manual describes the universe, cardinality, elements, and valid values for the collection. The data is collected at the individual candidate level and held by programs in their data systems. Once the elements are defined, the Data Group, which includes the Data Stewards from each program, determines how the elements will be counted. All completed data manuals brought to the Board for adoption will have met the expectations (below) for creating a data manual for implementation.

Process for Development

Proposed data manuals are brought to the Board at its May meeting. Staff work with content experts, the Data Group, and the Metrics Group, when requested, to create a data manual that meets the implementation requirements through vetting with the Data Group. The developed data manuals are brought to the Board for adoption the following May and include the elements contained in the proposed version from the previous May, along with any supplemental elements proposed by the Metrics Group and that meet the data element requirements of the Data Group. All elements proposed by the Metrics Group will have been collectively agreed upon by the professional association’s process and have the commitment of all educator preparation programs to be collected by them.

The Data Manuals are intended to remain relatively static from year to year. Updates to the data manuals are driven by a Board policy or oversight need. Based on the type of change to be made, substantive changes revert to a content expert group, and technical changes are referred to the Data Group, or a subset, for resolution. Modifications to a Data Manual that are beyond technical fixes, such as adding or removing an element, are at the direction of the Board. The Board retains authority to direct staff to create or change data manuals based on the mandates or, in the case of collective action by the preparation programs, from recommendation by the Metrics Group to the Board for inclusion of data elements in a statewide data collection.

Educator preparation programs will have the opportunity to review developed data manuals prior to the Board’s decision on whether to adopt the data manual. Board deliberation on the approval of the collection and the data manual will be open to public comment. General public comment is available at all other Board meetings, and written comment can be submitted to the Board at any time through the regular commenting process. Programs are invited to individually or collectively offer input into the cycle as provided by the PESB policy feedback protocols published here.

Expectations

Data elements in each DM will comport with the following requirements:

Operational

    • Include a unique code, definition, example, and list of valid values (where appropriate).
    • Prevent creating silos of information within programs.
    • Be granular enough to aggregate into multiple reports.
    • Have potential data collisions identified and addressed.

Measurable

    • Captures what actually happens rather than what is intended to happen.
    • Includes a timestamp that identifies when the element occurred.
    • Reflects variance beyond binary values.
    • Identifies relative rather than categorical values.
The Data Group will determine whether a data manual meets these expectations.

The Annual Reports home page links to examples of Data Manuals.

Report Guidance

Each year a Report Guidance document is developed by PESB staff. This document identifies the data to be reported to the Board.  The content is limited to elements defined in an approved Data Manual, as authorized by the Board, or available from PESB with proper linking keys. While the Data Manual defines the information programs collect on their candidates, the Report Guidance defines the aggregations of that candidate level data that are to be submitted to the PESB. These aggregations may include joining elements from multiple Data Manuals as well as joining outside data (e.g. OSPI S-275) to the Data Manual elements. The Report Guidance defines the data universe, cardinality, elements, and valid values for the reports as well as the due dates for those submissions.

The Annual Reports home page links to examples of Report Guidance documents.

Process Benefits and Challenges

Programs report specific data annually to the PESB. While all of the report contents derive from the data manuals, not all of the report information is used only by PESB. Some of the data is batch uploaded for federal reports, such as the HEA Title II teacher preparation program report. This practice provides efficiency for programs by reporting similar data a single time to a single body and allows them to retain responsibility for holding the data of record in their files. Common definitions across reports, resulting from the work of the Data Group and content experts reduce confusion and create consistency across all reports, state and federal.

This data collection and reporting process has challenged program level data system capacity. All recognize the benefits of commonly defined data efficiently collected and securely held, but designing systems to achieve these benefits, including connecting between the program and larger university, or even within the programs themselves, has proven more challenging for some than others. As the regulating body that defines the requirements for program to collect and hold specific data for reporting purposes, PESB has adjusted its resources to provide staff technical assistance. In addition, resources have been made available, as well, to help programs build data systems capacity through identification and support of technical assistance provided by peers.

For a description of the Data Project, please read the Data Project Background.