Step 1 Identifying a Problem

America has a labor shortage. The U.S. Chamber of Commerce states, "We are missing 1.7 million Americans from the workforce compared to February of 2020" (para. 1). So, not only is there a shortage for various reasons, such as retirement, changing careers, becoming homemakers, or entrepreneurs, but there is also a question of the number of prepared individuals entering college and the workforce. 

There are complaints from academics and businesses that people only sometimes have all the foundational skill sets needed to be successful in today's marketplace or to operate successfully in higher education. However, many products on the market suggest it can help suss out the best resumes or applications for hiring managers or admission committees to review and simultaneously provide job seekers or students with an understanding of their readiness for college or career. Just take a look at a list on LinkedIn.

According to Intelligent.com, which surveyed 1,243 business leaders, 40% believe recent college grads need to prepare for the workforce and list work ethic and [lack of] communication skills as the top reasons these individuals are not workforce ready. In addition, those who do not finish high school statistically perform lower than their high school counterparts. Couple this with the low proficiency rates in Math and Reading across the country, and it is clear that people struggle with these essential skills from grade school through high school and into college and the workforce. 

Point of fact, ZoomWV, an educational dashboard that averages the test scores for children in West Virginia (WV), grades 3, 4, 5, 6, 8, and 11th grades, shows that ten years ago, 2014-2015, ~70% of WV students were not proficient in math and ~53% were not proficient in reading. These are the same kids that have entered the workforce since their testing. Reviewing the 2022-23 school year, ~64% of WV youth are not proficient in Math, and 56% are not proficient in English in grades 3, 4, 5, 6, 7, 8, and 11th grades. Further, the Census Bureau's Quick Facts West Virginia shows that ~12% of "over 25 years old" do not have a high school degree, and only ~22% have a bachelor's degree.  

So not only did WV youth graduate high school at a rate of ~88%, but it would be fair to say, given the proficiency scores in math and reading, that the majority was not workforce or college-ready. 

These statistics provide correlative evidence that can help explain the incompletion rate of college-goers. The national college dropout rate is ~24% within the first year and ~32% overall. West Virginia was higher than the national average, and the figures did not include students who died, were totally and permanently disabled, or left school to serve in the armed forces, foreign aid service, or church missions. If the student dropped out and did not notify their reporting institution, it was not counted in these numbers. 

Without a high school diploma or college degree, one way to show prospective employers and committees an individual, at a glance, is workforce-ready is through a system of "digital badging." These resources are becoming a standard to help verify what is on a person's resume or application. In other words, an organization can call to check past employment history, references, and education, but it is more difficult to determine if the skill set is authentic. 

However, job seekers are not the only people wanting to understand or gauge their skill knowledge; those who wish to obtain education post-high school and did not make high grades, or dropped out and took High School Equivalency Test (HiSET), or went to college for a while and did not do well, or someone that has been out of school for a while and desire to understand what they have retained or want to learn things in "bite-sized" pieces. Micro-credential services cost a lot less than formalized education and provide the consumer a lot more choice in what is expected of them and how they interface with the content and provider; as such, they are becoming more popular as an alternative to more traditional learning approaches.

For job seekers and those wanting to learn or affirm skills, digital badges can be displayed and signify the holder has obtained, through assessment, a specific type of knowledge or skill(s). Some badges are also e-certified. In other words, the issuer verifies the badge as legitimate, usually through a registration or tracking number. A badge is awarded after a person passes an evaluative assessment or exam. We wanted to know how well test takers from prominent badging services like ACT WorkKey Digital Badgings faired.  

ACT is a nationally recognized household name in education testing. It is a brand of educational institutions trusted to measure what the student has learned to determine their academic readiness for college. But, they also have an instrument, ACT WorkKeys Solutions, which "is a system of assessments, curriculum, and skills profiling that determine, build, and measure essential workplace skills that can affect ...job performance and increase opportunities for career changes and advancement. (ACT WorkKeys – Workforce Development Solutions, para 1)." So, we set out to review it further. 

Figure 1. Percentage of ACT WorkKeys Examinees Meeting WorkKeys Skill Levels 

The illustration above shows a breakdown or distribution of the percentage of people who scored at what level per type of test from June 2017 to June 2019. This revealed most people, given the total number of test takers with each assessment scored the following: 

~28%/250,704 people scored a level 4 or Silver in Math

~36%/325,490 people in Workplace Documents

~34%/299,122 people scored a Level 5 or Gold in Graphic Literacy. 

The following is the total number of test takers per assessment:

• 895,374 Applied Math testers

• 904,141 Workplace Documents testers

• 879,773 Graphic Literacy testers

The ACT National Career Readiness Certificate (NCRC) level is given to a tester based on their score. The score levels are Platinum, Gold, Silver, or Bronze. The level is determined by the lowest score taken among the Applied Math, Workplace Documents, and Graphic Literacy assessments.  Level Scores: Platinum- 6; Gold- 5; Silver- 4; Bronze- 3. 

Levels 1 and 2 are not tallied because "an individual who took all three assessments and failed to achieve a minimum score of Level 3 on one or more of the three assessments would be considered an NCRC non-qualifier (Lefebvre et al., p. 5)." 

Lefebvre et al. provided in Table 7 the US Mean Wages for Jobs by WorkKeys scores, so at the lowest "Bronze" level, the mean wage for a test taker was $33, 586 in Math, $28,953 with Workplace Documents, and the lowest wage being $28,418 with Graphic Literacy. On the other hand, the highest wage was in graphic literacy at $130,132. 

The mean wage amounts make sense, considering the lower educational attainment, the lower the wage. See Quick Facts, West Virginia. There is a correlation between earnings and educational attainment, as shown below in graphs #10 and #11. Referencing that persons listed as "No high school" have earnings of $20.3 and those listed with a "Graduate degree" have earnings of $54.7. 

b. Instructional Design Literature

Purdue University states, "The principles of instructional design considers how educational tools should be designed, created and delivered to any learning group, from grade school students to adult employees across all industry sectors (para. 2)." Instructional design effectively uses the methods and approaches to teaching while understanding the individual learning needs of people. 

When contemplating course design, Standford University states it is important to understand:

1. Why the course is being created.

2. What content will be created?

3. The support the learners will need.

Stanford University also suggests that to ensure the reliability of the course, one should use observation and feedback to inform the design process. For the adult learner, Stanford University profers the ADDIE format for instructional design. ADDIE provides a 5 step process: 1) Analysis, 2) Design, 3) Development, 4) Implementation, and 5) Evaluation. "In the ADDIE model, [as shown in the illustration below] each stage must be carried out in order and carefully considered before moving on. Reflection and feedback at each stage ensure continuous improvement. (Van Vulpen, para. 5)."

 ACT WorkKeys has created curricula as a component of its NCRC assessment process. The ACT WorkKeys curriculum is a validated and well-developed product, but it does not contain hands-on activities, mainly because it is an online product. Our philosophy is that hands-on experiential learning activities allow students to deepen their understanding of the content through discovery and exploration. 

Hands-on activities help students go beyond the written word and graphics, as in step 2 of the ADDIE framework: "Create a learning intervention..." Hands-on activities that rely on a student/learner-centered approach to teaching can engage learners, deepen understanding, help retain knowledge, and foster creative and critical thinking skills.  

The Knowledge Network for Innovations in Learning and Teaching (KNILT) states, "Hands-on learning (method) goes by a variety of names such as experiential learning, project-based learning, and learning-by-doing, but they all mean the same thing. (para. 2)." We would add it could also pertain to problem-based, active, and inquiry-based learning. Hands-on learning is active or direct participation (doing the learning) versus passive (receiving the learning). But, hands-on activities are not meant to be provided in a silo; the University of Sandiego offers a comprehensive list of teaching methods that illustrates the sheer number of ways hands-on activities, as a method of instruction, can be combined for greater impact. For instance, a lesson that utilizes a hands-on activity could include a brief lecture, group discussions, and a student portfolio to bring the hands-on experience together for a meaningful experience.

While relevant to the overall content, hands-on activities must also be "minds on" or purposeful, as described by the PEAR Institute through their Dimensions of Success (DoS) observational tool,  planning tool, and feedback and coaching tool. 'Minds-On refers to how the student, cognitively speaking, interacts with the activity. In other words, we want to avoid what is referred to by DoS as "cookie cutter" activities where you follow the steps without cognitive exercises, like making a cookie recipe: step 1: get the ingredients; step 2: mix the ingredients; step 3: bake it.  

For instance, in Applied Math, the first level after the introduction is  Counting • Numbers & Sequences • Addition • Subtraction • Multiplication • Division. At this level, using everyday items as manipulatives can help students see and experience basic math concepts. A teaching approach might be using manipulatives such as dice, cards, marbles, geometric shapes, Lego blocks, checkers, dominoes, etc, as part of games. 

Another approach could be conducting science experiments with educational supplies to discover and explore various mathematical relationships or using engineering activities to illustrate multiple mathematical constructs. Guiding students through discovery and exploration with science and engineering is an excellent pathway to understanding mathematical principles because math, in its varied forms (algebra, geometry, calculus, etc.), is the language of science, technology, and engineering.  

One could take an existing written curriculum with a particular format for its lessons and purposely add relevant hands-on/minds-on activities to bring the content to a purposefully active and wholly participatory level.

Step 3: Specifying a Purpose and Research Questions

We have written a purpose statement, research question, and hypothesis to provide critical direction to this study. 

Background.

To earn the ACT WorkKeys National Career Readiness Certificate (NCRC), one must pass three assessment exams: Applied Math, Graphic Literacy, and Workplace Documents. 

What are our initial questions?

Based on the test-takers' results with the ACT WorkKeys NCRC in West Virginia, we are curious if test-taker scores could be improved by adding hands-on activities to the existing ACT WorkKeys Curriculum and ACT WorkKeys Curriculum Details (time and effort). 

Research Question.

Can we improve NCRC test-taker scores by providing  Project Lead The Way (PLTW) hands-on engineering activities in conjunction with the ACT WorkKeys online curricula, compared to the validated cohort that did not receive intervention?

Hypothesis

Based on the available literature, we should be able to assist test takers in improving their scores nominally from their baseline test by utilizing PLTW hands-on activities in the program. We do not expect a significant change compared to the validated cohort. 

Step 4: Designing a Study

a. Methodology

For the hands-on learning portion of our project, we will use Project Lead the Way Engineering activities. Simultaneously, students will participate in the interactive online WorkKeys Curriculum covering Applied Math, Graphic Literacy and Workplace Documents. 

We simply want to learn if adding the hands-on instruction improves their scores compared to the validated cohort.  Each cohort member will be assigned a number corresponding to the WorkKeys online curriculum Account for assessment testing. Regarding instructional design, we will use ADDIE as the framework. 

b. Milestones/Timeline

Issues:

The hiring freeze, federal review, and lacklustre registration with student and teacher recruitment drove us to make an administrative decision to embed the pilot project in a high school engineering and math course.  Below reflects the original and updated timeline.

Post Award Timeline:

• January 2024- Request a money transfer. Begin creating the website. 

• February 2024- Trello Board creation.

• March 2024- Received transfer. Request a purchase order for online curriculum fees. 

• July 2024- Gain access to the online curriculum.  Request that the ACT Assessment contract be signed (In July 2025, still trying to obtain contract signature).

• July 2024- Dec 2024: Work on curriculum enhancement activities. Develop activities around learning centers based on interests/intellects.

• August 2024- Form a collaborative team for program efforts, meeting weekly.

• October 2024- Trello board updates

• November 2024- Timeline and Schedule, Form and letter creation, speaker/tour inquiries, research, IRB Consent

• January 2025- Work on IRB pre-requisite material gathering

• February 2025- Start teacher hiring process (put off due to federal review and subsequent hiring freeze)

• March 2025- Curriculum and supplies are ready (curriculum was readied in April, and supplies were put on hold)

• April 2025- Registration release and Teacher request (Summer program cancelled due to lacklustre registration and teacher acceptance efforts)

• May 2025 - Order Supplies (put on hold due to summer program cancellation) 

• June 2025- Garner IRB approval (the IRB request was forwarded in June after administrative pre-approval)

• June 2025-Implement  (postponed until fall 2025- plan to embed in high school engineering and math program)

• August 2025 - readying training for upcoming term.

• September  2025 - implementation in 9th-12th grade elective course- Engineering.

Teacher Training Timeline

Will occur online, as needed throughout implementation.

c. Schedule

Academic Inclusion Schedule 2025-2026

The Lab days will be set up in multiple centers to accommodate people's interests and intellects. The Office Hours will be online only, allowing students to pop in and out to discuss with the instructor and, if warranted, go to breakout rooms to help mentor one another. 

• Orientation Day  | Sign IRB

• Pre-Engineering-Project Lead The Way (1-year program)  The Pre-Engineering Program of Study focuses on a broad range of engineering careers and foundation knowledge, including basic safety, plan reading, and the use of tools and equipment, as well as how to employ positive work ethics in an engineering career.

• Test Day  

• Graduation Day

• Tour Day A

Time Per Day: Academic Afternoon Schedule 26 weeks 

Step 6: Reporting the Research

Step 7: Implications for Practitioners

Step 8: Follow-Up Questions

Does the timing of the intervention matter?  In other words, which is a more effective measure for NCRC test-takers, "aligned activities" or  "gap activities"? A gap activity is one where the activity is given after the test is taken, and a problem is identified - it is treated as an opportunity to rectify a knowledge gap. On the other hand, an alignment activity is one where the activity is given before the test - it is treated as an embedded part of the curriculum. Further, what role does subject matter play in the efficacy of either of these measures? In other words, would we find test takers in Workplace documents benefit more through gap activities, whereas test takers of Math benefit with aligned activities?

Bibliography

“ACT WorkKeys – Workforce Development Solutions.” ACT, 18 Jan. 2024, www.act.org/content/act/en/products-and-services/act-workkeys.html. Accessed 19 Jan. 2024.

Denes, Melissa (January 17, 2005). "'Feminism? It's hardly begun'". The Guardian. London. Archived from the original on November 9, 2014. Retrieved November 16, 2014. 

Ferguson, Stephanie . “Understanding America’s Labor Shortage.” U.S. Chamber of Commerce, 7 Feb. 2022, www.uschamber.com/workforce/understanding-americas-labor-shortage.