Vocal Jazz for U.P. - EVALUATION

List of Contents on this Page:

1. A word on a good student evaluation of teaching vis-a-vis the Kirkpatrick model

2. Detailed Evaluation of Day 1 iteration based on the Kirkpatrick Model

Level 1 Reaction
- Post-Day 1 iteration survey with participants
- Post -Day 1 iteration survey from instructor/SME/facilitator
Level 2 Evaluation- Learning
- Detailed song recording evaluation notes;
- Lis' comparative performances in focus
Level 3 Evaluation – Transfer (Proposed short-term evaluation)
Level 4 Evaluation – Results (Proposed long-term evaluation)

Note: It is presumed that you have read through the HOME, ANALYSIS, DESIGN, DEVELOPMENT, and IMPLEMENTATION pages prior to reading content on this page. Thank you.

THE KIRKPATRICK EVALUATION MODEL (Kirkpatrick & Kirkpatrick, 2016)

Important Note:

"In a good SET [student evaluation of teaching] instrument, a set of 10–15 questions will have excellent content validity–i.e. the ques-tions collectively measure essential aspects of preparation and teaching–and highreliability–i.e. student judgements are consistent and inter-related positively acrossitems (Newell et al.2004). A variety of recent studies provide useful guidelines fordeveloping SETs (Alok2011; Bangert2006; Barnes et al.2008; Denson, Loveday,and Dalton2010; Lemos et al.2011; Spooren, Mortelmans, and Denekens2007)...The margin of error is the index of choice in survey research when making an inferenceabout the population mean from sampled data. We previously defined the interpretativevalidity of a score as the precision of the interpretation given a score in a particularassessment situation. Most survey research texts in the social sciences recommend amargin of error equivalent to plus or minus 3% of the total range of the scale (Dillman,Smyth, and Christian2009;Fowler2009)" (James et al., 2015).

Detailed Evaluation based on the Kirkpatrick Model

Level 1- Evaluation – Reaction (iterated) (See Appendix 3.21)

Post-Day 1 iteration survey with participants

The survey had 10 questions in total. Everyone answered the survey. Response rate was 100%, as survey responses were inputted via email right before the Day 1 session ended. With a minimal margin of error conjectured, I can say with a 90 - 95 per cent confidence level that the MuEd 105 batch of 2022-2023 who joined the beta iteration of the workshop showed positive, promising dispositions (Agree-Strongly Agree) toward vocal jazz content, vocal jazz as an idiom, learning to sing vocal jazz further utilizing bel canto technique, and having a masterclass with a jazz practitioner. An area for improvement perceived would be the students' ability to differentiate between Jazz and Classical treatments of Summertime within the session. Learner-teacher interaction/engagement also appeared to be a challenge.

MuEd 105 students who joined the beta iteration of the vocal jazz workshop strongly agree that they are interested to learn to sing standard jazz repertoire and believe that vocal jazz has to be part of formal academic curriculum.

Post -Day 1 iteration survey from instructor/SME/facilitator

Ma'am Sherla agreed to:

"I felt comfortable teaching/facilitating the Day 1 of the vocal jazz workshop at least from a bel canto-trained teacher's perspective

I think I was able to engage the students in the activities- like making them differentiate between Classical and Jazz vocals based on vowel formation etc.

I like the idea that the workshop is sort of gamified- meaning participation in Day 1's activities would earn the students/participants a badge to the following days' sessions";

whereas strongly agreed to:

"Two students-Lis and Archi- were able to do contrasting recordings (Classical and Jazz techniques) of a song of their choice and able to submit the recordings via email. I believe this fulfilled a major objective to Day 1 of the workshop.

I believe it is highly possible to create an official vocal jazz workshop at the U.P. College of Music that will introduce bel canto-trained singers to vocal jazz

Introducing bel canto-trained/ Classical singers to Jazz can widen their artistic and creative knowledge and enrich their performance acumen"

Ma'am Sherla is confident that the objectives for Day 1 were achieved, and that it is highly possible for UP to have a vocal jazz workshop. Also, she strongly agrees that a vocal jazz workshop can benefit MuEd 105 students in terms of broadening their artistic and professional acumen. Her personal recommendation is for a practitioner that performs in both Jazz and Classical genres be invited to help further refine the workshop.

Level 2- Evaluation – Learning (Appendix 3.22)

Rubrics-based evaluation of learning (vocal jazz singing in focus) at https://docs.google.com/document/d/1EXtYF4jwKTnV5D_Wrg4C0jZJ4Wbmeo3FuMsGo6tO_aI/edit?usp=sharing

Evaluation rubrics are based on the Ward-Steinman (2008) study involving evaluation of Australian and American university jazz singers, as well as a previous jazz voice assessment guide created for Berklee Online's Jazz Voice course in EDS 157 (Lacdao, 2020).

Notes on Day 1 iteration jazz song performances

Lis provided a Classical (bel canto) recording and a Jazz recording of the same song in English--- treatments were supposed to be contrasting. There was noticeable difference in the jazz treatment--- namely a more bouncy vibrato, more palpable note-bending though subtly executed, and the vocal texture's tendency to be huskier. Using assessment rubrics, Lis earned a score of 2.8 for the jazz recording (between basic and standard). This performance is acceptable given that Day 1 does not focus on straight-ahead jazz yet.

Archi provided a Classical (bel canto) recording and a Jazz recording of the same song in French--- treatments were supposed to be contrasting. The Classical song was delivered in standard bel canto fashion (dome-shaped vowels, overbearing vibrato and a seeming vertical movement/migration of tones across cavities/resonators). His jazz treatment of the same song was interestingly different- he swung the notes and removed his vibrato. There was a great effort at trying to syncopate the lyrics. Archi scored 3.36 for the jazz recording (more than standard).

Average jazz song performance score for entire population during beta iteration: 3.08 (more than standard).

Note: Both students scored high (more than standard) in the Tonal Substance criterion, whereas scoring low (basic to standard) in the Rhythm and Creativity criteria. These results/ratings/observations alongside the song recordings will be forwarded eventually to the Day 3 masterclass provider for a learner needs analysis prior to students joining Day 3.

Detailed song recording evaluation notes at

https://docs.google.com/document/d/1FVzt51ZXZHZ6Ts9Sq6WhljIkZftSgS75jsJJEqA-Hws/edit?usp=sharing

The case of Lis

Lis had provided a jazz song performance from the previous e-workshop (iSpring Suite e-workshop in Feb 2023). Her recording then was rated at 1.3.

In the following beta iteration in May 2023, she obtained a higher score of 2.8. This means an increase by 73.1707%.

Therefore, the expectation is that on average, in the Day 2 and Day 3 performances for an upcoming Version 2.0 iteration, an improvement of between 60-70% should be expected from Lis' 2.8 score on Day 1 of beta iteration. A score of approximately 3.75 - 4 should be expected from participants on Day 2, and 4- 4.25 on Day 3 (masterclass)

Links to Lis' song recordings:

iSpring Suite-based e-workshop in February 2023

Jazz recording: https://vocaroo.com/1oJ9NDgZI8Bi

Beta Iteration in March 2023

Classical: https://vocaroo.com/15xEpSs5vetJ

Jazz: https://vocaroo.com/1iUOGytKyNEp

Based on Lis' two successive vocal jazz performances with an improvement/increase of approximately 70%, it is then expected that vocal jazz recordings from Day 1 through Day 3 must have a positive increase of around 60 - 70% in between.

Level 3- Evaluation – Transfer (Proposed short-term)

Performance on Day 3 during the Masterclass- must reach a score of at least 4 based on the rubrics-based assessment. This should happen within less than 1-2 months from Day 2.

Referencing Lis' comparative performances above, therefore, the expectation is that on average, in the Day 2 and Day 3 performances for an upcoming Version 2.0 iteration, an improvement of between 60-70% should be expected from Lis' 2.8 score on Day 1 of beta iteration. A score of approximately 3.75 - 4 should be expected from participants on Day 2, and 4- 4.25 on Day 3 (masterclass).

Important Note:

• Teacher (masterclass provider) assesses the performance using above-mentioned rubrics. Expected score is at least 4. A score of 4 will earn the learner a certificate of basic vocal jazz competency to be provided by the masterclass pedagogue.

• A score of at least 4.25 can merit the badge for today- to sing a jazz song on RJTV as well as an opportunity to sing in Japan or New York with Lorna Cifra’s vocal school. Will also merit a certificate of basic vocal jazz competency.

• Only a student with at least 4.25 can get both the badge and certificate of competency

After the masterclass, participant students have to answer a survey:

https://forms.gle/ox5Lg4yMSZGFtm2r7

Level 4- Evaluation – Results (Proposed long-term evaluation)

Those graduates who have taken the workshop (say batch 2022- 2023 MuEd 105)- must be interviewed one to two years after graduation if they were able to utilize their learnings from the vocal jazz workshop- did they get a jazz gig? Were they able to perform in jazz venues? Did they take up master's programs related to vocal jazz?

A cohort study may be conducted- interviews with previous MuEd 105 students who had not taken up the jazz workshop versus those who have taken the jazz workshop. A comparison can be made in terms of employability in the music business, ability to take on jobs that require versatility etc.

Also, it is interesting to note that Lis and Archi graduated this year 2023 from their Bachelor of Music (Major in Voice) programs magna cum laude (Pharaway Lacdao, personal communication, July 2023). This could be a good research question- How does taking a vocal jazz workshop help a MuEd 105 student with academics? Does it help with becoming more creative? Does it help in crafting more interesting lesson plans for teaching voice? Does it help expand the range of the voice as a Classical singer? Does jazz make a singer better at listening/audiation in musical terms? A qualitative case study can be applicable to this particular goal.