Final Project Report

 Executive Summary

Dr. Martha Doran, an associate professor in the School of Accountancy at San Diego State University, has created and delivered two online modules for course Accounting 421 – Auditing. She wanted to know how well her modules worked and if she could improve it. The professor wanted the student instructional consultant to help evaluating current online modules, developing three new online modules, and proposing recommendations for continuing improvement and further study. During the project, a needs and satisfaction survey was conducted to collect information for actual situations; communication via Google Docs clarified the vision, goals, and characteristics of students; analysis of the survey identified opportunities for improvement; and such design and development improvements of online modules were completed as well as a post-class survey was conducted for the final evaluation.         


Introduction

This paper described a semester-long practicum project for ED795A at San Diego State University involving analysis, design, development, and evaluation of the professor’s five online modules from two blended courses for auditing in accounting.

The course Accounting 421 (Acctg. 421) contained two portions: auditing and advanced financial accounting. This project focused only on the auditing section. Acctg. 421 – Auditing and Acctg. 626 - Auditing were courses that introduce what, why, and how a professional CPA (certified public accountant) audited financial reports. The professor has taught the course for many years. This year, she decided to make a change to course delivery format by transferring twenty percent of the face-to-face classroom-teaching course to web-based online teaching. She chose three out of fifteen modules from each course to go online. The main consideration for her selection was to give flexibility to students’ schedule for attending classes in-person. With a long history of interesting in computer-assisted learning, the professor had intrinsic motivation to make these audacious changes. According to the professor, the positive feedback from two previous online modules also contributed to her decision to continue exploring new alternative learning methods.

SDSU administrators encouraged teaching staff to use more online learning in order to free more classroom space. The infrastructure for such teaching was already implemented for use. For professors, the main tasks were to design and deliver the online modules, up to fifty percent of a course, on a reliable online learning management system - Blackboard.

Since the project began with two existing online modules, analysis of satisfaction and needs must first be completed to collect information about current teaching conditions. On the other end, evaluation after implementation was essential for knowing how well the new intervention has done and what questions could be further studied.

The challenge for me was to use theories as a framework to solve real problems systematically in a very short time.

Purpose

The purpose of the project was to evaluate current online modules, design and develop new online modules, and to provide recommendation for future research or improvement as a student consultant.

The project was funded by the School of Accountancy—one of several programs within the College of Business Administration. The major stakeholders were the professor and the students as well as College of Business Administration which has been trying to get more faculty members to develop online curriculum. The professor invested time to discuss, design, and develop materials with me as the student consultant. The students invested time and effort on the modules and in the survey. Both the professor and the students were affected by the learning outcomes.

The targeted performance outcomes were:

·     Design online modules that will engage more students.

·     Facilitate the professor to build structural view about online learning.

·     Evaluate two existing modules, analyzing findings, and proposing specific criteria for evaluation.

·     Suggest questions for future improvement and studies to the professor.

After an initial understanding of this project, I learned that I was expected to play three of the several performance consultant roles that Hale (2007) so aptly details: the critic, the doer, and the spectator. Although I do not have the expertise in accounting to make discipline-specific suggestions, I am a professional of instructional and performance technology whom my client counts on to tell her what was good and bad. I was also a doer who was responsible for analysis, design and development. Moreover, I was neither the professor nor a student, I was expected to observe the process, record the comments from the professor and students, recommend possible solutions, and facilitate the participants toward improvements. I decided that the first thing to do was to conduct analysis of the current instruction: getting to know the characteristics of the audience, the actual teaching situation, and the optimal vision expected.

Impact of the Literature and Course Materials

The review of literature has focused on articles and research papers regarding blended learning, online course evaluation, and instructional design. The review identified the importance of the following:

·     Clear strategy for separating what content should go online and what content to stay in-class. (Fransen & Swager, 2006; Valiathan 2002): Re-examining course goals and objectives, considering which would be better achieved online and which would be better achieved in face-to-face class. 

·       Effective management of cognitive workload of audiences (Clark, Nguyen, & Sweller, 2005, p. 45): using multimedia to explain complex content, especially when the learners are new to the topic.

·       Engaging online learning activities and asynchronous discussion (University of Wisconsin-Milwaukee, 2008).

·       Clear evaluation criteria for continuing improvement (Ellis & Calvo, 2007)

·       On time and on demand learner supports (Mactague, 2004).

Methodology

The methodology used in this project, as shown in the figure below, aligned with the instructional system design model, ADDIE. The model begins from front-end analysis including understanding audiences, describing the optimal, identifying opportunities for improvement and causes for preventing success. Then, continue with design, prototyping, and developing the online modules. After implementation, the cycle ends in evaluation.

Communication

Since the professor and I were located in different time zones, communication has been challenging throughout the project. Most communication was done asynchronously via email and Google Docs. For synchronous communication we used Adobe Connect Professional, the computer conferencing tool, and Skype, the instant messenger. When there was a need for immediate conversation, we discussed over Skype; when audio and text communication could not explain our meaning, we created computer conference rooms using Adobe Connect Professional in which we integrated graphics and slides to visualize abstract points.

To keep the professor informed of my progress, I created a time logs file that recorded tasks I have completed each day on Google Docs. If she had questions about my works, she would leave notes on the document and I would get back to her very quick. Both the professor and I were very responsive; therefore, time difference didn’t become a barrier of the project.

Understanding the Audience

Information of the audiences was mainly provided by the professor as audience analyzer. Only a small portion of it was gained via survey. Students of the classes were 83 accounting major undergraduate students who have taken some classes on advance financial accounting (three students dropped out the course later in the semester) and 33 graduate students with various educational backgrounds. 97% of the undergraduate students were in their senior year. They were on campus students with no or very few experiences in online learning. The class met twice a week for three hours each meeting. Before taking online modules, they have met each other in the classroom for advanced finance instruction.

Describing the Optimal

During the first meeting, I found the professor had difficulties in describing what an ideal online module should look like. Therefore, I provided the professor with three evaluation tools from Quality Matters, EduTools, and California State University, Chico, and asked her to choose one tool that, she thought, would fit best in her case and could be a good tools for evaluating existed online modules as well as a good guideline for analyzing and designing new modules. She selected the tool from California State University, Chico (the Chico tool). It was reasonable to assume that the exemplary performances described in the Chico tool described the ideal online module in the professor’s mind. The table below described the requirements of an exemplary online course in the Chico tool.

Category

Description

Learner Support & Resources

Course contains extensive information and resources; provides various course-specific resources; displays contact information for instructor, department, and program; offers access to a wide range of resources.

Online Organization & Design

Course is well-organized and easy to navigate; syllabus delineates the role the online environment will play; aesthetic design of the course communicates course information clearly; web pages are consistent; accessibility issues are addressed.

Instructional Design & Delivery

Course offers opportunities for student-instructor, student-student, student-content interaction; goals and objectives are clear and aligned with each other; learning activities integrated with objectives; provides activities for students with different learning styles.

Assessment & Evaluation of Student Learning

Course has multiple timely and appropriate activities to assess student; aligns learning objectives with instructional and assessment activities; uses multiple assessment strategies to measure content knowledge, attitudes and skills; provides regular and timely feedback about student performance.

Innovative Teaching with Technology

Course uses a variety of technology tools to facilitate communication and learning; uses new teaching methods to enhance learning and interaction; uses multimedia learning objectives and/or elements; optimizes Internet access and effectively engages students.

Faculty Use of Student Feedback

Instructor offers multiple opportunities for students to give feedback on course content, online technology, and ongoing assessment.

Identifying opportunities for Improvement

I developed and administered a needs and satisfaction survey – a mix of closed- and open-ended questions. The instructor encouraged student to take the survey by providing extra credit. The return ratio was high. Seventy-six out of eighty students (95%) responded to the survey. By analyzing the frequency distributions and central tendency of the survey results, I obtained a clearer picture about the strength and weakness of and the students’ reaction on previous two online modules. The answers to open-ended questions also provided valuable information for both satisfaction and needs on new online modules.

In addition to the survey, I interviewed the instructor (as the subject matter expert and the incumbent of the project) and the teaching assistant (as the colleague of the project) via instant message and/or email to learn more about the challenges they have encountered as well as the intentions and expectations. In addition, I reviewed two sets of extant materials: 1) students assignments submitted to Digital Drop Box in Blackboard to gain some ideas about the learning results; 2) course content on Blackboard website and textbook website.

Comparing the current situation (data gathered from survey, interview, and extant literature) to exemplary performances described in the Chico tool, I identified the following major gaps.

Category

“Optimal” Performances

Opportunity to Improve

Course overview/ introduction

·    Course syllabus identifies & clearly delineates the role the online environment will play in the total course.

·    Course goals are clearly defined and aligned to learning objectives.

·    Learning objectives are identified and learning activities are clearly integrated.

·    Around 48% students said they needed to check back and forth between several sources to understand course syllabus (about goals, learning results, learning objectives, activities, requirements, assignments, assessments, and textbook website)

·    How to meet specific instructional objectives are not clear to 83% students.

Course design & delivery

·    Course offers ample opportunities for interaction and communication student to student, student to instructor and student to content.

·    Course provides multiple visual, textual, kinesthetic and/or auditory activities enhance student learning.

·    Course provides multiple activities that help students develop critical thinking and problem-solving skills.

·    No instructor-student or student-student interaction provided for online modules.

·    Course activities were mostly textual activities.

·    Learning activities provided on textbook website had the potential to develop students’ critical thinking and problem-solving skills; however, the activities would likely be performed through student-content interactions in independent study hours by student alone.

Assessment & evaluation

·    Course has multiple timely and appropriate activities to assess student readiness for course content and mode of delivery. Learning objectives, instructional and assessment activities are closely aligned.

·    Ongoing multiple assessment strategies are used to measure content knowledge, attitudes and skills.

·    Regular feedback about student performance is provided in a timely manner throughout the course.

·    Students’ self-assessments and peer feedback opportunities exist throughout the course.

·    The first online module has no student readiness pre-assessment.

·    Assessments provided on textbook website provided multiple assessments strategies; however, most of the assessments are used to measure knowledge and skills.

·    Regular and timely feedbacks were rare to none for previous online modules.

·    Peer review were absent from previous online modules.

·    Grading criteria were not transparent to students. Grading results were not posted timely.

Learner support & resources

·    Course contains extensive information about being an online learner and links to campus resources.

·    Course provides a variety of course-specific resources, contact information for instructor, department and program.

·    Course offers access to a wide range of resources supporting course content.

·    Student readiness support about being a successful online learner was not provided.

·    Student readiness support for course-specific prior knowledge was not provided.

·    Student readiness supports for technologies required, such as ACL software and Digital Drop-Box, were not provided.

Identifying Causes for Preventing Success

Purpose for blended learning was not clear to all participants

The reasons for moving classes online could be plenty. For administrators, the reasons might be ‘free more classroom space’, ‘easier to manage’, or ‘higher ROI (return on investment)’. For instructors, the purpose for going online has to be more than “answering to the request of the department”. Good reasons for integrating online instruction could be “increasing the quality, consistency, and accessibility of training”; ”maximizing content sharing and reuse”; and ”increasing learning effectiveness and throughput by institutionalizing best practices” (Office for Domestic Preparedness, & National Domestic Preparedness Consortium, n.d.) What the instructor stated in the interview was also a good reason for online instruction: “To shift more of the conceptual, knowledge based activities to the individual on-line method and then use the in-person class time for more group activities such as role simulations, group cases, presentations by groups, etc.” However, it seemed that the instructor’s view on purpose was not shared and discussed with the teaching assistant and students. The consequences were, students questioned, why they needed to learn online and the teaching assistant did not like the new change on submitting assignments online so much. If students and the teaching assistant were aware of the benefits that online learning could bring them and how they were going to try a new environment progressively, they might be able to endure the imperfection raised during the process and benefit from a promising learning environment. They might even help improving the temporary weakness and defects.

Strategies for blended learning were not defined

What is the right mix of online and in-classroom modules? What topics to select for online? When is the best timing for starting the online learning practice? Is every element either in-person or online, or students could make their own choices? (Fransen & Swager, 2007).When these questions were raised and thought over, the online module wouldn’t be merely a substitute of classroom teaching just for convenience. What kind of interaction could be applied online? What tool could be used to check students’ online learning outcomes? More ideas about fostering students’ competencies for creation, evaluation, and analysis should be discussed and realized.

Format for online learning was vague

Were the existing two online modules designed for self-paced or collaborative learning? Different formats have different mixes of content, activities, assessments, and feedback systems. If the formats were defined in advance, students would better understand what to expect from online modules (on-time help, real-person interaction, returning time, etc.), and the instructional designer would easily recognize any additional elements that need to be included.

Interactive activities were missing

Interactive activities could encourage students to think more critically. It was not likely that high-level thinking, such as judgment skills expected by the professor, would take place via online learning when student-instructor and student-student interaction was missing.

File management was not considered

While navigating the Digital Drop Box section in Blackboard, I saw hundreds of assignment files stored there with no organization. It would be very difficult for one teaching assistant to provide timely, meaningful feedback when she needed to grade 116 copies of each assignment. File management might be a small issue but it could induce a nightmare.

Student support system were not established

Much dissatisfaction resulted from students’ low readiness for online learning. When students were not prepared for learning in the new online environment, new technologies might confuse them and new challenging disciplines would make them even more anxious.

Design the Online Modules

Based on the analysis, the design considerations include:

·    Add hyperlinks to connect related content together

·    Use multimedia to reduce cognitive load and ease off feeling isolated

·    Design interactive activities to increase instructor-student and student-student interaction 

·    Provide technology support and prerequisite knowledge support

·    Enrich assessment methods to include peer review

·    Elaborate points of difficulty by audio.

 

Developing the Online Modules

Since the online modules were built on Blackboard, an online learning management system, development was a matter of clicks on the Blackboard menu. First, the professor sent electronic files of related content to me. Then, I uploaded the files to different sections on Blackboard. After proofreading and simple pilot testing, the new content was made available to students. For podcasting, the professor used Blackboard embedded tool, Wimba, to record her audio. As soon as the recording was completed, the podcast was ready for download by students immediately.

Limitations and Constraints

A major limitation for this project was the time constrain. There were only five weeks between the date I joined the project and the date the first online module was released to students. During the five weeks, I accomplished many tasks including front-end analysis, interviewing the professor, designing the questionnaire and conducting the first survey which entailed collecting data, analyzing data, interpreting the results, writing and submitting the analysis report to the professor, discussing changes to be made with the professor, developing module content, proofreading, reporting defects, making modification and releasing the final product. Some design ideas were not realized due to the limitation of time.

Evaluation

After students took the new designed online modules, both classes completed post-class surveys. The results showed that the senior accounting major students in Acctg. 421 and the non-accounting major graduate students in Acctg. 626 had different impression on the online modules. The return ratios were 92.5% for Acctg. 421 and 72.3% for Acctg. 626. The overall impression from students was positive. Several students made remarks that the new modules were ‘much better’, ‘easier to follow’ or ‘significantly improved’ than previous modules.

Student support and resources

·    Students in Acctg. 421 class were clearer about where to find the links to other resources than students in Acctg. 626 class. This was because the new module was the third online class for Acctg. 421 but was the first online class for Acctg. 626. A note worthy point was that 52.8% of students in Acctg. 421 was still confused about what areas of the websites they were required to visit, even with two previous experiences in taking online modules.

·    55.4% of students in Acctg. 421 and 29.2% in Acctg. 626 have accessibility problems with technologies used. The problems were mainly about using ACL. Students mentioned this but we did not address the problem in any new module due to time constrain.

Instructional design and delivery

·    62.5% of students in Acctg. 626 were somewhat or very interested in having recaps on prerequisite knowledge online. This percentage was higher than the ratio showed in Acctg. 421. The reason should have connection with students’ educational background. Only one student in Acctg. 421 was not an accounting major; but only one student in Acctg. 626 was an accounting major. However, even for the students majoring in accounting that were enrolled in Acctg. 421, 55.4% of them expressed their interests in having links for related knowledge on hand.

·    Most of the module-specific contents including multimedia and text materials were greatly appreciated by students in both classes. On a 5-point scale, 14 out of 21 items have a rating average of 4 points or higher in Acctg. 626 class. Only 2 items, the Drop-Box and Discussion Forum, were rated less than 3.5 points. In Acctg. 421 class, 9 out of 21 items have a rating average of 4 points or higher. Only 1 item, the Discussion Forum, was rated less then 3.5 points. Among all the content, students considered MCQ solutions, Homework solutions, and examination study guides to be the most helpful. Students were obviously very score-oriented. Although the professor did provide a tutorial for Drop-Box technology in new modules, students still had trouble using it. The reason might be the audio tutorial provided by the professor did not come with visual aids such as graphics or video; therefore, students could not follow the guides of the audio tutorial.

Interactivity

Interactivity appeared to be the weak point of all three online modules. When asked about the flexibility of choosing learning methods, assignment topics, and discussion topics, the mean for the three questions was 2.92 for Acctg. 626 and 2.83 for Acctg. 421 on a 4-point scale. Timely feedback on students’ work was low, 2.8 for Acctg. 626 and 2.85 for Acctg. 421. Several students remarked ‘very slow’, ‘were not answered’, or ‘no feedback’. However, when asked about responsiveness of faculty, the rates were an average of 3.2 points, higher than the rates for ‘Timely feedback’. This was a little conflicting with each other. As to the quality of feedback, either classes rated 3.1 or lower on a 4-point scale.

Assessment and evaluation

·    Have the modules improved students’ critical thinking and problem-solving skills? 62.5% students in Acctg. 626 and 39.2% students in Acctg. 421 “strongly agree”. Another 45.9% students in Acctg. 421 said, “somewhat agree”.

·    Is the assignment workload too high? The replies were fifty-fifty; half the students claimed that the workload was too heavy while the other half students considered the workload adequate.

·    Only 12.5% and 20.3% students in Acctg 626 and 421 said the instructor knew their level of readiness. Pre-assessment were not conducted. Some researches suggested that pre-assessment would be helpful for a large class teaching.

Overall impression

In final comments, students made several valuable suggestions. Some of them concurred with my suggestions made for previous modules, but have not yet been addressed in the newly designed modules:

·         Providing sample essays. Students said, “It would have helped. . . to have essay assignment details and know that outline was required prior to the module. . .”

·         Using the discussion board. Students said, “I think they would be enhanced with some discussion board interaction with other students and the instructor.”

·         Needing “better coordinated with the financial module”; “Switching from one professor to another can be confusing”, students commented.

·         “Doing the survey right after the assignment”, students said.

·         “How long will it take for studying one online module?” some students raised the question. They suggested, “Give the estimate amount of time to complete the online module. . .” When students knew the time they would need to spend in the online modules, they would be more ready for the challenge rather than assuming that the online module will be easier or the online module will save hours for study. As long as students have the right expectation, complains will be reduced.

Recommendations

Faculty Development and Learner Support

Previous study identified that faculty and student preparation for success in teaching and learning in blended programs is one of the key factors that lead to success (Voos, 2003). Faculties needed more than technical skills to make the transition from face-to-face teaching to hybrid teaching. Faculties must rethink and redesign their courses and new learning activities. They also need to learn teaching skills for online facilitation and new assessment methods using technologies. A synchronous or asynchronous, face-to-face or online, a faculty development program will be helpful.

Learner support is as important as faculty support. Tutorials for technologies used, recaps for relevant knowledge, sample products for completing online assignments, etc. all contribute to reduce learner anxiety and increase student satisfaction.

Define a Process for Creating New Online Content

In order to be effective in preparing online teaching, defining a process to ensure that all key factors are considered is necessary. Resources for guidelines and tutorials for hybrid teaching were plenty. Hybrid Courses developed by University of Wisconsin-Milwaukee is one good resource and Teach Online from Michigan State University is another. However, every course has its own unique characteristics and the instructor needs to tailor the process to her/his own needs.

Questions for Further Study

On completion of the project, there are several questions left for further study:

·    Effectiveness. When re-examine instructional objectives of the course, which would be better achieved online? What is the right blending in terms of learning effectiveness?

·    Connectivity. How will the work done in online modules or face-to-face classes support and connect with each other? How will two portions instructed by two professors of one course be connected with each other?

·    Engagement. For many students, online learning means they work on their own and are isolated from classmates. A breakthrough for the myth is learner engagement. Online asynchronous discussion and interactive activities could greatly draw in participants. What are challenges and solutions for using these?

Conclusions

I like the on-line sessions. I wish there were more of them”.
I like the online classes because you get to be hands on
”.
(I like online teaching), it forces them (students) to think about the sites we visit and give their ideas, which in class not everyone does this, since there are so many students in a class”.

Above feedbacks from students and the professor clearly demonstrate that online learning is an approach worth exploring, especially when blended with traditional classroom learning. Blended instruction can be a door to begin exploiting the potential of technology in improving the quality of learning and teaching for traditional classroom instruction.

I believe that the professor and I have learned from and enjoyed the process even though we spent considerably more time than expected. We both further recognize that, creating online modules of a course is not simply transferring a portion of traditional course to the web. Instead, it involves a series of tasks concerning strategy, pedagogy, learning activity, assessment, and techniques.

In terms of EDTEC competencies, this project experience has built or broadened my skills in technical, cognitive, communication, and interpersonal areas.

Comments