Executive Summary
Mary and Michael have been asked to evaluate the successfulness of their company’s Corporate University. In order to do this, they have both decided to use Appreciative Inquiry techniques which they have both studied previously, but have never actually used. The Board of Directors has given them eight weeks to do this entire process and report back to them their findings. The purpose of their evaluation is to determine the effectiveness of the company’s Corporate University.
They believe that the most effective way to conduct an evaluation from their research is by using Appreciative Inquiry(AI). From their findings, AI will give them more positive feedback and less negative feedback which they believe will result in a more productive evaluation.
To create the evaluation, they first created a program logic model. This is similar to a road map – it helps them keep track of their progress during the evaluation and analyze their findings. In the creation of the model, they identified their key stakeholders and formulated three key questions for the evaluation. These questions are:
1) What technology(ies) have you been using during your Corporate University courses that you have found useful in other areas of your life (work or personal)?
2) How has your experience at the Corporate University impacted your life (work or personal)?
3) Can you give us at least one example of how the Corporate University has positively impacted your life?
In order to collect the data required, Mary and Michael have decided to create a focus group comprise of students from Corporate University. These stakeholders will then be asked to attend a meeting where they will be asked to work together to answer several different questions, some AI some not, in order to collect the data desired. Once the focus group meeting has been concluded, they are planning on presenting the results to the Board no earlier than two weeks following the meeting.
The overall evaluation plan has been outlined so that the evaluators know what is planned and everything runs smoothly. The focus group is to include 12 students – former and current, along with two evaluators. We believe that this size of group is not to large to stay focused and will give us good baseline information.
In order for the focus group to work, they have field tested their evaluations, which are both included for your viewing. They believe that all of the work they have done researching AI and creating the evaluations will result in the information that the Board has requested. They do realize that this is a daunting task however and will still take a lot of work to compile their final results.
For Mary and Michael I have three recommendations:
1. Time – I would recommend they be given more time to not only create but also conduct and prepare their final report for their boss and the board of directors.
2. Budget – Give them a formal written budget amount.
3. Create an evaluation schedule – Create an overall planed evaluation schedule for the company. Not just a one time evaluation.
I would also suggest that they look into not only using the focus group but also setting up an evaluation survey that could be sent to more members. This would allow more data to be collected from a larger population and may give them additional feedback that may be useful in the evolution of the Corporate University.
Situation
Mary, Vice President of Human Resources, and Michael, Training Director, have been asked by their boss to help prepare a presentation for the Board of Directors of their company about the impact on the company of the newly created Corporate University. They have been given very little information, except that they need to keep their boss apprised of the situation, and that they need to complete this evaluation and presentation within the next seven weeks. The reason that Mary and Michael have been asked to create the evaluations is because they have taken courses on evaluation techniques in the past. The boss and the Board of Directors want to use this educational experience to help out the company. Mary and Michael are a little uncertain how they will be able to do this evaluation in a manner that meets all the needs and is credible in such a short time span however. This is why they are first planning out what they need to do and including it as part of their presentation, so that they can hopefully show what needs to be done, and the time it will take to complete a thorough evaluation of the Corporate University in order to gain useful and credible feedback to be used.
Purpose of Evaluation:
The purpose of this evaluation is to determine the value and effectiveness of the Corporate University within the corporation. The Board of Directors want to know if the money being spent to create, maintain, and deliver the courses is a fiscally sound decision for the overall organization.
Why & How AI Will Be Incorporated:
Mary and Michael are still totally unsure at this point how they will completely evaluate the Corporate University. However, they have both been doing research into evaluation techniques and have both read different articles and research on Appreciative Inquiry. They believe that if they use some of these techniques for their evaluations that they will receive not only more input about the Corporate University, but information that will be much more substantial and more useable information. They believe that it will give them more subjective information for the evaluation.
They want to make sure that they do no harm and that they are using the Program Evaluation Standards to do everything in an ethical manner so that their evaluation results are credible and are not something that will later be questioned. They also believe that this will give them a better picture of how the Corporate University fits into the entire corporation. They are looking at using AI in order to gain information that is not the typical negative feedback, but to also help the participants think about what is working within the organization. They feel by doing this, they will get a better overall picture of what is happening.
Evaluation Plan:
In order to gain the information that is needed, a Program Logic Model Mary and Michael have decided to create this in order to help them focus on what needs to be done in order to gain the information that they feel the Board of Directors is looking for. It will also allow them to have a plan on what they need to be doing in order to get the desired information. In order to create a Program Logic Model, they first started by looking at different logic models online. Above is the Program Logic Model that they have created to help them with their evaluation plan.
Stakeholders
The Program Logic Model that Mary and Michal have created, indicates that information will be gathered from the following stakeholders: staff and students. Additional stakeholders include the Board of Directors and upper-level management. Findings from the evaluation and subsequent decisions made regarding the Corporate University will affect all stakeholders to some degree.
Key Questions
Three key evaluation questions will be examined during the evaluation process. These questions are:
4) What technology(ies) have you been using during your Corporate University courses that you have found useful in other areas of your life (work or personal) (KQ1)
5) How has your experience at the Corporate University impacted your life (work or personal)? (KQ2)
6) Can you give us at least one example of how the Corporate University has positively impacted your life? (KQ3)
Data Collection, Analysis Methods, Timeline & Other Plans
A focus group meeting will be held to collect data. A select group of students of the Corporate University will be invited to participate in the workshop and asked to provide their input. Appreciative Inquiry methods will be used to help participants get to know each other so as to feel comfortable and to help increase participants willingness to provide answers to the key evaluation questions.
Stakeholders will work together in different groups throughout the day in order to obtain as much information as possible and as many differing perspectives about what is happening within the Corporate University as possible. At the conclusion of the workshop, information that has been collected from the small groups will be reviewed by the larger group to ensure that everyone is aware of what information has been gathered and to verify the accuracy of the information that has been recorded during the day.
Evaluators will then review and compile the information that was gained from the focus group and create a report summarizing the results.
Summary bullet points from the initial draft of the report will be sent to all participants in the focus group so that they may have a copy to see what will be given to Board of Directors and the overall findings. The entire process should take approximately 2 weeks.
Overall Evaluation Plan
We plan on holding a focus group meeting where we spend 60-90 minutes (90 max) gaining information to help us evaluate the effectiveness of the Corporate University. After focus group meeting, information with be compiled copies sent to group members to make sure information is accurate and so that they are also able to see results of meeting and the evaluators’ findings.
Location
We will hold our focus group meeting in a location that is easily accessible to everyone. Issues we will be looking at when selecting the location include the size of the room, we don’t want to be crowded, but also don’t need excessive room. Making sure it is a comfortable location for all. We want our group members to be comfortable and not feel as if they are unable to share information because where they are.
Focus Group
12 participants – 6 current & 6 former students if possible
Evaluators – 2 – one to conduct meeting & one to take notes/record findings
Rules
Give a copy of these along with paired down version of the agenda as meeting begins so everyone knows exactly what is/is not expected out of meeting.
Basic Meeting Agenda
Icebreaker/Interview Questions:
Discussion
Mary and Michael have been given the daunting task of creating and conducting an evaluation for their company’s Corporate University. In order to conduct this evaluation, they have been looking at many different aspects of what makes an evaluation successful and what is the best way to conduct an evaluation. After undertaking this background research, they decided to use Appreciative Inquiry to conduct their evaluation.
In order for them to create their AI evaluation, they created a program logic model and an evaluation instrument. Their pre-field tested instrument can be seen in Appendix A of this paper. Their post-test field instrument has been included above. As one can see, there have been a few modification made from the pre-field tested evaluation to the one that will ultimately be used in the evaluation.
Lessons Learned From Project
Many lessons have been learned from this project. The main lesson that I have learned is what Appreciative Inquiry is and how it can be used in any situation if the evaluators are willing to put in the time and effort to create an AI evaluation. I have also learned that in order to create any effective evaluation tool, it not only takes time but planning. You must have an idea of what data you want to get out of the evaluation. If you ask very basic yes/no types of questions, you will get limited information but not any additional detail that you may also want. If you use AI questions, you are able to get more thoughtful answers from participants. It may be more difficult to get organization members unfamiliar with AI that are helping with the evaluation process to agree to creating and conducting an AI evaluation, however, if you can explain the benefits of AI, it may be easier in the future to get buy-in for using an AI approach to evaluation.
My Recommendations
I have the following recommendations for Mary and Michael:
1. Time – I would recommend they be given more time to not only create but also conduct and prepare their final report for their boss and the board of directors.
2. Budget – Give them a formal written budget amount. By letting them know in advance what money they have to spend will allow them better plan what they are going to do or not do with their evaluations.
3. Create an evaluation schedule – Create an overall planed evaluation schedule for the company. By creating a set schedule then they will be able to track the progress of the Corporate University. This will also allow them to have better insight as to what needs to be done within Corporate University. If they begin now, then this will give then baseline information that can then be referred to as more data is collected in the future.
Overall, I would strongly recommend that Mary and Michael proceed with their evaluation plan using Appreciative Inquiry. I would suggest however that they may want to make their evaluation be a two pronged system. One prong being comprised on the AI focus group. They can possibly have different groups for each program possibly within Corporate University. The second prong would be an evaluation. This would have AI questions, but also be some basic questions that one may see on a typical paper survey. By creating this system, they would be able to gain information from a larger population of employees which may be helpful even as to why others may not be using Corporate University.
Appendix A: Pre-Tested Field Instrument
Corporate University Interview Protocol, Questions, and Information
Overall Evaluation Plan
Hold at focus group meeting where we spend 60-90 minutes (90 max) gaining information to help us evaluate the effectiveness of the Corporate University. After focus group meeting, information with be compiled copies sent to group members to make sure information is accurate and so that they are also able to see results of meeting and the evaluators’ findings.
Focus Group
12 participants – 6 current & 6 former students if possible
Evaluators – 2 – one to conduct meeting & one to take notes/record findings
Rules
Give a copy of these along with paired down version of the agenda as meeting begins so everyone knows exactly what is/is not expected out of meeting.
Basic Meeting Agenda
Icebreaker/Interview Questions:
i. While groups are presenting, make sure to see if any other the other groups need clarification on what is/is not said by other groups. Give 5 minutes for each group compile answers for each question before having them present answers to group.
References:
Migotsky, C. Conducting Focus Groups video. Located at https://d2l.global.uillinois.edu/d2l/orgTools/ouHome/ouHome.asp?ou=9553&contentURL=/lms/discussions/admin/forum_topics_list.d2l.
Preskill, H., & Catsambas, T. T. (2006). Reframing evaluation through appreciative inquiry. Thousand Oaks, CA: Sage.
Stevenson, H. (2009). Appreciative Inquiry: Tapping into the river of positive possibilities. Retrieved on June 11 from http://www.exinfm.com/pdffiles/Appreciative_InquiryTapping_into_the_River_of.pdf
Voyle & Voyle Consulting (2009). Appreciative interview guide for organizations. Retrieved on June 10 from http://www.clergyleadership.com/ai-resources/AI-generic-ques-org.pdf