The Operational Technology Assessment (OTA) 301 course at XYZ Inc., a pseudonym, focuses on operational technology systems and processes and hands-on training for practical application in the field. Based on Kirkpatrick's Four Levels of Evaluation model, Mr. K., an instructional designer at XYZ Inc., is overseeing a Level-3 Evaluation program for the OTA 301 course.

The Level-3 Evaluation of the OTA 301 course is a pilot program consisting of a 16-question survey for participants who completed the course in August 2023.The mixed-methods survey includes both closed and open-ended questions. Mr. K. has plans for follow-up interviews and focus groups. Through this survey, the Level-3 Evaluation aim to explore how participants are using OTA concepts in their jobs. It's purpose is to determine the effectiveness of the course in real-world situations—in other words, the application of concepts and behavior change on the job. The findings will be used to inform adjustments for improvement.

Our team set out to evaluate the Level-3 Evaluation program to determine if it was effectively measuring on-the-job behavior change or new skill application. We also wanted to evaluate the overall evaluation design, including data collection methods and sources. Working with the client, we developed a program logic model (PLM) that outlined resources, activities, outputs, and outcomes. It's important to note that the PLM was created for the OTA 301 course, not specifically for the Level-3 Evaluation, to provide context for our team's understanding of what was being evaluated.

Several stakeholders were considered when conducting this evaluation:

Upstream stakeholders: two instructional designers, two project managers, and government and private funders.

Direct Impactees: past, current, and future participants and technical staff members.

Indirect Impactees: Due to the program's role in national security, participant success impacts social, economic, political, and security domains at a national level.

Evaluation Purpose and Type

The Level-3 Evaluation was still in its pilot format, so our evaluation was designed as a formative evaluation to assess the Level 3 evaluation efforts of the OTA 301 course, aiming to identify strengths and weaknesses. It was goal-based, as our team was determining the extent to which the Level-3 Evaluation program was achieving its goal of measuring participants' behavior change.

Dimensions, Evaluation Questions, and Importance Weighting

After determining two dimensions based on stakeholder needs, Kirkpatrick model standards, and expert consultation, our goal was to evaluate data collection methods (weighted very important), while also assessing environmental factors (weighted important) that support or hinder the evaluation. When evaluating dimension 1: data collection methods, our guiding questions were around methods, sources and design. For dimension 2: environmental factors, we posed the questions around organizational and cultural factors that would impede or be driving factors for the Level-3 Evaluation.

The evaluation focused on two critical dimensions: data collection methods and environmental factors. The breakdown is as follows:

Identified project limitations, risks, and ethical considerations shaped the evaluation trajectory. Security constraints prevented the environmental survey, revealing barriers for future projects. Despite limitations, evidence-based practices were applied, including benchmarking against peer-reviewed research and industry literature, utilizing rubrics, and incorporating expert feedback.

Despite limitations, we adhered to evidence-based practices, resulting in an ethical and valuable evaluation. The report aligns with client and organizational goals, maintaining professionalism and ethical standards throughout the 10-step evaluation process. While recognizing constraints, the report provides a comprehensive assessment, enabling XYZ Inc. to make informed decisions for the Level 3 evaluation program's refinement.

In conclusion, while the OTA 301 Level-3 Evaluation, at this pilot stage, holds potential, implementing recommendations will strengthen the program.

To improve the Level 3 evaluation program, we proposed the following recommendations: 

* Enhance Data Collection Methods:

* Required questions in the self-administered survey will provide evaluators with deeper insights.

* Timing Adjustment: send out the survey 1-3 months post-course to allow enough time for more accurate assessments.

* Diversify Data Collection Methods and Sources:

* On-the-Job Observations: introduce workplace observations for a holistic understanding of participant application.

* Virtual Simulations: explore realistic virtual simulations as an alternative, given facility restrictions. 

* Expand Feedback Loops:

* Participant Feedback: establish feedback loops through interview and/or focus groups to gain qualitative insights.

* Peer and Supervisor Feedback: ask for comprehensive views from peers, direct reports, and supervisors.

In implementing these recommendations, the Level 3 evaluation of OTA 301 can refine its strategy and instruments, showcasing efficacy, identifying areas for enhancement, and ensuring continued support and funding. 

This was an interesting project to complete as our team was essentially acting as meta-evaluators for the Level-3 Evaluation of the OTA 301 course. With our evaluand being in its beginning stage, we had to conduct a bit of an opportunity assessment or front-end analysis, as well as a formative evaluation to come up with recommendations that would make this evaluation stronger. As we also had our professor and peers evaluating our evaluation, it became a bit of a house of mirrors, where we kept evaluating evaluation principles and best practices, which I think was what made this project such a great educational experience.