developing effective Design Challenges

DEVELOPING EFFECTIVE DESIGN CHALLENGES 319

Iteration Beginning With a Prototype Design

Few middle school students have had the experiences with technology that help them believe they can design and build working devices, algorithms, and systems.¹¹ We have found that by initiating challenges with a step-by-step “cookbook” start-up design (such as an electromagnet with 20 wraps of #18 wire around an 8p nail), a much larger fraction of students become immediately absorbed in the activ­ity. Diagrams and teacher demonstrations allow students with absolutely no back­ground in construction to succeed in this initial but critical step. We have found that all students can reproduce a simple working model, albeit a poorly working one. Middle school students appear highly motivated to best what they perceive as their teacher’s design, although the initial “prototype design” for each challenge is care­fully crafted to be both easy to build and instructive to improve. This technique has proven especially helpful for students with few manual skills. For example, our two-dimensional “paper truss” challenge begins with a sheet of notebook paper suspended by two of its holes, easily supporting a 1-kg weight from a hole in its base. Students have no difficulty in constructing improved designs by incrementally cutting away paper in search of a lightweight truss. Students do vary in the degree of risk with which they are comfortable, some cautious, others rash. Both approaches have their value and liabilities, which do not escape student atten­tion, just as in the design endeavor as practiced in society.

We find that students usually prefer to work on initial challenges individually to build their self-confidence, usually side-by-side with their friends. They quickly move to small groups, especially when allowed to help each other. This method replicates the way design is often carried out in the world of work. Engineering is usually practiced in teams, but teamwork only becomes productive after individu­als develop skills and self-confidence (Hatfield, 1990). The fruits of increased self-confidence became apparent as students began working on their projects out­side of class. Working devices became trophies that were demonstrated to friends at recess and lunch. Students then brought their optimized device home to show their families and friends, often receiving expressions of wonder and words of en­couragement. Devices began to flood in from home (e.g., broken doorbells and small electric motors for the electromagnet module) for study.

We have found the use of a prototype design to be more effective than starting from scratch even when students are not building devices. For example, as an as­sessment activity, we compared students who were given the task of presenting ev­idence that their batteries improved as a result of their activities. Half of the students discussed the elements of a good graph and then tried to produce one that showed their findings. They were compared to another group that started with a poorly conceived “prototype graph” with three scattered points. Students begin­ning with the prototype graph were quick to identify its limitations. They began the task with confidence and stayed with it far longer than the group that began with the more conceptual approach. The prototype design group demanded less of the teacher’s time, discussed their work more with others, incorporated other groups’ data more often, and produced a more diverse set of representations of patterns. For many students, theory helps to organize their ideas only after they have had a sufficient number of concrete experiences.

We tested the impact of using an initial prototype with 10 dyads of sixth grade students designing and building electromagnets (Schwartz, 1998). Initially, the teacher demonstrated to students the assembly of a prototype design electromag­net. Lifting a single paper clip demonstrated that the electromagnet worked, but did not reveal how well. The instructor then posed the challenge: “Do you believe you could improve this design?” Students suggested strategies and the teacher noted those suggested changes on the board for future reference (e.g., number of nails, size of nail, number of wraps, etc.). This step established the potential vari­ables students could investigate. Students were encouraged to only change one variable at a time.

The dyads were then split into two groups. One required to build and test the prototype before embarking on their own designs and the other group was free to start in any fashion they wished. Examining the work produced by the two differ­ent groups shows a difference in their ability to change one variable at a time with the prototype group succeeding in holding all variables but one constant for 80% of their attempts, while the non- prototype group succeeded only 53% of the time. A t test comparing the 10 dyads shows the differences as significant at the p £ .05 level.

Perceptual control theory (Powers, 1973) argues that the goal must be clear enough to learners that they can first envision ways to achieve the goal, and second be able to interpret the feedback that their actions generate. Those dyads starting with a prototype design are reinforced in their pursuit of identifying variables by having an unambiguous reference with which to compare their later designs. To the extent that teachers wish to model how scientists use controls in their experi­ments, the prototype design does scaffold this objective.

Using a prototype design rather than a pure discovery approach helps to support students in working at a functional level higher than that of which they might oth­erwise be capable (Fischer & Pipp, 1984). Venturing into a new level of abstrac­tion, that of discovering the scientific principles governing a device’s performance, is aided by having students able to physically handle and examine concrete manifestations of their ideas. We like to have students build and save their designs whenever possible, so that the objects can be held, examined, talked about, and shared with others. Rather than having only data surviving from an ex-periment, preserving the concrete object helps students to concentrate on the ab­straction of differing performance due to an underlying principle.

The opportunity to perform many iterations is very helpful for students. Be­cause of the short iteration time, students can test many ideas thereby developing confidence and honing their building techniques. Also, we have seen no evidence of “dry-labbing” (falsification of test results) that students engage in when more conventional laboratory experiments do not perform as expected. A short time be­tween iterations serves to reduce the “ego investment” by students in their designs. When a device does not perform as expected, they appear less likely to blame themselves and, in a more healthy and constructive fashion, blame the idea for not working. Design-and-build challenges at higher grade levels are typified by spend­ing weeks on a design and testing them only once. We find a mixture of small suc­cess and failures keeps student engaged and productive.

Purposeful Record Keeping

Conveying one’s ideas and issues to others is critical in the modern workplace. We have found ways to facilitate communication by planning projects so that coopera­tion and communication are encouraged and rewarded. All tests of devices and pro­cesses in DESIGNS are public; others can watch and learn from what they observe. We have experimented with personal laboratory notebooks to record progress and ideas, finding that few students seem to see any intrinsic value in careful record keeping. Only when these records repeatedly became of use during reflective activ­ities did we observe a gain in popularity and a serious increase in record-keeping activities.

A powerful reflective technique for students is the making of storyboards. The storyboard first originated in the Disney studios as a way to outline the evolution of a cartoon without committing to all the details necessary in the finished product (Denison, 1995). Student storyboards are used to document the story of how a challenge was met over time. It is not created at the beginning of the project. Nei­ther is it created at the end of a project as a summary of what was accomplished. The storyboard is a series of frames created by students during the project, each frame displaying the latest solution to the challenge. Starting with the prototype design, students typically create three to five frames (of drawings and data) that tell the story of their investigation (Figure 9). These storyboards serve a quite dif­ferent purpose than “lab reports” in that they document the sometimes curious routes to discovery and include predictions, interim results, insights, and failures. The storyboard provides a pictorial as well as a literal database of student progress in trying to meet the goal of improvement. Student devices are either attached to or drawn in each frame.

FIGURE 9 Solar shelter storyboard. Here a team of two keeps a record of their experiences during two challenges. For the top three cells the students progress by adding aluminum foil strips to reflect the lamp’s rays, reducing the temperature by 1 °C from the prototype design. A subsequent sealing of the “attic” space is predicted to be useful, but does not lower the tempera­ture of the house.

Holding variables constant is not a natural strategy for students (Schwartz, 1998). They perceive such an approach as too wasteful or slow, wanting in­stead to try many ideas simultaneously. It is only when the storyboard is re­flected on, after a construction is complete, that a student can be directed to focus on defining exactly which change in design produced the claimed im­provement. Students from other teams often exhibit healthy skepticism, point­ing out alternative hypotheses for changes in performance. Those storyboards that show a single variable change ultimately receive the recognition for a dis­covery. Students learn how easy it is to fool themselves into believing that the wrong variable was responsible for a change. The richness and visual nature of the storyboard record helps students to draw new knowledge from the data they have collected.

Storyboards become respected evidence for claims of discovery. In some cases (such as electromagnets) the actual devices can be fastened to the storyboard, add­ing authenticity to the record. In other cases, the development of drawing skills, construction of flow charts, or graphing competence become well documented and a subject for student reflection. Teachers find storyboards a useful focus for reflec­tive study, asking: How did student drawings change? Did they more clearly repre­sent the device? Which test was most productive? Which change could the student have done without? If the student were to draw a fifth frame, what would be changed and why?

CONCLUSION

Engineering challenges are viable alternatives to free exploration activities and tra­ditional laboratory experiments for middle school science students. Although orig­inally utilized at higher academic levels, these challenges can be adapted to meet the abilities and interests of middle school students. We have found that special at­tention must be paid to student goals and student preconceptions in designing such challenges. Students must “buy into” the goal of the activity; they must understand what is expected of them or they will flounder. Moreover, they must be able to uti­lize the feedback that results from testing their ideas. The design challenge must throw into stark contrast the prior beliefs of the student and the science concepts that we wish them to learn. Through iteration and public tests these concepts are discovered and found productive. These design tasks can also motivate attention to accurate record keeping.

Through research in 20 classrooms we have been able to identify several attrib­utes of design challenges that are effective with school children in Grades 5 through 9. Student designs should be tested and modified often; there should be at least one iteration by each team of students within each class period (45–55 min). These tests should not be made explicitly against each other’s designs, but against nature, measuring with stopwatch, scale, ruler, or thermometer. Students should set their sights on learning to control nature, not on outperforming another team. Students delight in their own improvements relative to where they started, some progressing gradually, others in fits and starts. Teachers should expect and point out a variety of attitudes toward risk, and failure should be acknowledged as a fail­ure of ideas, not of people. All tests should be conducted at public “test stations.” Although this may cause a bottleneck upon occasion, students learn important les­sons from observing the performance of others’ designs, among them novel ideas that they may later choose to incorporate or neglect in their own designs, and is­sues of testing in a uniform and fair manner.

Design challenges should possess an intrinsically large dynamic range in per­formance of a single measure; there must be lots of room to improve. For students to find their prior conceptions lacking and be willing to adopt a new idea, the evi­dence must be overwhelming. We have aimed for a 10× improvement in perfor­mance as an attainable goal for students and continue to modify our challenges to increase this range. Our latest record involves building a loudspeaker from scratch. We are able to attain a gain from +1 db over ambient noise (60 db) to +50 db (110 db). This represents a 100,000× gain in efficiency (10⁵).

Starting with an easy-to-build but poorly functioning “prototype design” ap­pears to offer great advantages to students who have few prior experiences in de­signing and building. Constructing an initial functional “cookbook” design, no matter how poorly it performs, results in a feeling of accomplishment for students that helps to propel them toward investing in improvements. Utilizing compari-sons to this common starting point also helps while discussing the improvements and failures that teams experience, and aids students by having concrete manifes­tations of abstract principles. Without the set procedure of most traditional labora­tory experiences and the prior knowledge of expected results, we have seen little falsification of data or copying of other’s results.

The need to utilize the data from many trials requires that students pay close at­tention to formative record keeping. Alternatives to traditional laboratory reports have arisen that preserve student ideas and the results of tests. The many iterations involved in each challenge have inspired journal-like storyboards that are con­structed during, not after, the week-long design challenge.

Our design challenges contribute to a growth in science process skills and in students’ realization of the unique aspects of the scientific process. Uncovering the causal links between changing parameters and the resulting performance demands that students discover how to vary one thing at a time. Trials proving that an idea that does not work can be more valuable than finding a change that does improve performance. Good record keeping is essential in settling disputes concerning who had which idea first. Replicability of results by more than one team adds credibil­ity to claims. Paradigm shifts abound as major discoveries are made and sweep through the classroom.

Rather than preserve the initial advantage of those students with prior build­ing experience, design challenges help students develop skills in planning, con­struction, and testing. Although many female students first appear at a disadvantage in these challenges, they soon learn the necessary skills. When competitions go on long enough, they often challenge the male students, show­ing how the playing field can be leveled through thoughtful changes in the mid­dle school science curriculum.

ELEMENTS OF SUCCESSFUL DESIGN CHALLENGES

IN MIDDLE SCHOOL

Clear goals: Challenges should reveal to students the exact nature of what is be­ing asked of them. Challenges should invite students to chose (or to consider) strat­egies they feel appropriate to attain the goal.

Tests against nature: Designs should be evaluated using highly reliable tests against nature and not rely on complex rubrics or subjective judgments of teachers or students.

Prototype design: Students vary in their construction skills and level of confi­dence. Building an initial “cookbook” design, albeit a poor performer, is a neces­sary first step to engage students, develop rudimentary construction skills, and familiarize students with test procedures.

Multiple iterations: Students learn from their failures as well as successes. To encourage the testing of ideas, devices should be quick to build and modify so that many tests can be performed in a short period.

Large dynamic range: Whenever possible, device performance should increase dramatically over several days of building. A high signal-to-noise ratio is neces­sary for students to find experiments and their data convincing and to uncover the underlying science.

Employ purposeful record keeping: Student records should be formative, cap­turing all attempts and trials. They need to function as a resource for the resolution of claims of first ideas and for the focus of class discussions.

ACKNOWLEDGMENT

This work has been supported by the National Science Foundation (ESI–9452767 and ESI–9730469) with thanks to Dr. Gerhard Salinger for his feedback and con­structive ideas.

Special thanks to Kristen Newton for her extensive review of the literature, Ju­dith Peritz for manuscript preparation, Pam Sears for research and editing, and Su­san Roudebush for project management expertise. Marcus Leiberman and Annette Trenga conducted evaluation activities. Additional members of the project team—Jay Hines, Kerry Rasmussen, Steve Saxenian, and Marti Lynes—aided in classroom trials. We thank our DESIGNS teachers: Stephen Adams, Marilyn Benim, Anne Brown, Nancy Cianchetta, Cynthia Crockett, Carolyn Fretz, Mary Ann Guerin, Anton Gulovsen, Kimberly Hoffman, Teresa Jimarez, Paul D. Jones, David Jurewicz, James Kaiser, Milton Kop, Laura Kretschmar, Barbara Lee, James MacNeil, Linda Maston McMurry, Daniel Monahan, Sarah Napier, Doug Prime, Diana Stiefbold, and Mary Trabulsi for their innovative ideas and creative teaching.

This article draws heavily from our National Science Foundation proposals of 1995 and 1998 and internal project reports.

REFERENCES

Beaton, A. E., Martin, M. O., Mullis, T. V. S., Gonzalez, E. J., Smith, T. A., & Kelly, D. C. (1996). Sci­ence achievement in the middle school years: IEA’s Third International Mathematics and Science Study (TIMSS). Boston: Boston College Center for the Study of Testing, Evaluation, and Educa­tional Policy.

Brandenburger, A. M., & Nalebuff, B. J. (1996). Co-opetition. New York: Doubleday.

Cohen, M. R., & Harper, E. T. (1991). Student-as-scientist and scientist-as-student: Changing models for learning from experience. Teaching Education, 3(2), 31–40.