Efficacy of Web-based Instruction

This post is in response to a recent report I read, The Efficacy of Web-Based Instruction at York University: A Case Study of Modes of Reasoning, 1730. The report was done by J. Paul Grayson, Suzanne E. MacDonald, and Jean Saindon (2001) at the Atkinson Faculty of Liberal and Professional Studies York University, Toronto.

The focus of this paper is on the academic achievement and course satisfaction of students taking

Modes of Reasoning 1730, which centered on critical reasoning skills. The course was offered by the same instructor in two sections, one in class, and one over the web. Of interest was whether the academic performance of web students would be comparable to that of traditional in-class students, after adjustments were made for prior levels of achievement and learning styles. Results showed that grades of in-class students on every form of evaluation were higher than those of web students, even after adjustments had been made for prior grades and learning styles (pg. 1).

The major problem of their study is that it focuses on technology which in and of itself does NOT enhance learning. Effective learning on the web or in a classroom is dependent on the skills of the instructor to adapt to the needs of the students and the design of the instruction regardless of the technology involved (chalkboards, TV, computers, etc.....).

I think that it is ironic that the article deals with the efficacy of web-based instruction. What about the efficacy of traditional classroom which is accepted as the de facto standard and is always used for baseline comparisons.

As Downes and others have pointed out, the problem with comparative studies like this one is that they measure the affordances of new technology based on what they could achieve using traditional methods. It's like comparing what type of communication is possible with email vs. regular snail mail. What you can do with email is completely different than what can be accomplished with snail mail (i.e. attaching audio/ video files, mass distribution, immediate feedback, linking to relevant resources. Each form of communication has its advantages and disadvantages which are relatively self-evident. Neither is perfect so what is the point of comparing the two?

As I read the article it came as absolutely no surprise that the in-class students performed better on traditional testing methods than web-based students taking the same tests when both groups of students were exposed to traditional forms of instruction.

"Both sections had the same readings, essays, tests, and final exams, and the text of in-class lectures was posted on the web." (pg. 3) Those types of assessments reinforce the strategies that were used - ones which students are quite familiar with.

The teaching methods used represents a very limited range of instructional strategies. Perhaps a case-based model where the students could exercise the critical thinking skills they are supposed to be developing in the course would have been appropriate.

The web-based version of the course should not seek to replicate what occurs in the classroom, but should strive to achieve outcomes which are otherwise not possible, ie. engage students in deeper levels and different kinds of interaction because there is no time in class for everyone to provide input on a subject.

If anything web-instruction provides an opportunity to enhance learning styles because information can be represented and accessed in different ways. I witnessed an audio/ video conference between students at Carleton and Waterloo as they worked in real-time on a shared application to solve a problem. The cameras at the workstations were voice-activated and would point to individuals verbalizing their viewpoints.

How would one measure the 'value' of someone collaborating on a project with students from a different part of the world to make decisions collectively which represent a variety of perspectives they otherwise would not have been exposed to? To me that represents the true power of the technology: the ability to globalize learning. As anyone who has taken a course online will tell you, there is just as much you can learn if not even more from your fellow students than you can from the instructor. In addition you grow to understand different regional contexts for applying what is being learned. It also means you have an authentic audience in which to express yourself and not just a one way communication channel to the
instructor.

A case in point were the previous findings cited in the paper. "Importantly, in a post-course survey, web students reported more peer contact than the in-class group." (pg 7) Schutte (2000) speculated that the performance differences between the groups could be attributed to peer interaction rather than to the use of computer technology.

The study also does not account for the shift required by students in assuming greater responsibility for learning in a web-based environment. Some students do not prefer this because of the increased cognitive load required to process the information. They are forced to develop critical thinking skills to analyze information as opposed to having it spoon fed to them by an instructor.

As I have stated previously, web-based assessments should reflect what is possible in that learning environment. There is little value in research which points to the inability of online instruction to improve performance in traditional areas of assessment (multiple choice tests, essays).

I also think that the focus of the authors on the institutional impact on critical thinking skill development is misguided. The focus should be on the classroom/ web-based strategies which promote this skill.

The authors included the following quote, "Another limitation of previous research noted by the Institute for Higher Education Policy is inattention to the learning styles of students availing themselves of different modes of instructional delivery." (pg. 7)

What they are really saying is that the design of the instruction ie. lecture style on the web is incompatible with certain learning styles. It has nothing to do with the mode of delivery. Accommodations for different learning styles/ abilities can occur in any environment whether or not it actually takes place.

Evaluation Results - Pharmacology Learning Object

I recently finished compiling the evaluation results of the pharmacology learning object.

Method

The participants consisted of 2 health science faculty and 10 students from the CLOE consortium. I used the Learning Object Review Instrument (LORI), version 1.4 developed by Belfer, et al. (2002) to collect faculty's individual assessments of the quality of the pharmacology learning object. Faculty were asked to assess the learning object in 8 categories using a five point rating scale ranging from low to high and to provide a rationale for their score.

A survey with 25 questions was distributed to students in order to carry out a learning impact study based on their use of the learning object. The student questionnaire items were combined into four sub-scales: Learning Value; Value Added by the Learning Object; Usability of Learning Object; Usability of Technology

For a more detailed description select evaluation methodology.

Results

The results from the faculty evaluation indicate that the learning object scored high in the categories of content quality, motivation and instructor/ student guides. It was also rated very good to high in learning goal alignment, presentation design and reusability. Interaction/ usability and feedback/ adaptation received lower scores ranging from good to very good and satisfactory to very good respectively.

Student Questionnaire results appear in Table 3. These results show that 75.7% and 77.5% of the possible responses within the Learning Value subscale and the Value Added subscales, respectively, were in the Agree and Strongly Agree categories. The responses for Usability of the Learning Objects and Usability of the Technology were in the Agree and Strongly Agree categories 80.0% and 92.0% of the time, respectively.

Student comments

The responses to the following questions that expressed the most common themes were recorded.

What did you like about this LO?

"I like the use of animations. I am a visual learner so any type of diagram is helpful, but animations are sort of novel and made the module more interesting."

"I liked the way it was broken down into the simplest components so you could figure it out yourself even if you still weren't sure about the material. Also, the fact that it is always there and you don't have to go out and get a CD."

"The interactive process and the use of the graphs related to drug effect on the body."

How could this LO be improved?

"Need to make output continue to relate to variables chosen by the learner. Should label diagrams a bit better for use beyond the demos. Should be able to have 'type of drug' selection as well, not just specific drugs."

"Sounds, more use of animation to support use of formulas. Narration. Glossary. Drug dictionary to describe what it is for.

Please use this space for any further comments you’d like to make about the LO including clarification of any of your responses.

"I would not prefer just a lecture on the topic. I feel that a lecture and a seminar using the LO would work well together."

"The equations page was kind of useless for a basic knowledge of pharmacology. It doesn't tell you what it's for and how it is tied in."

"I found that the LO improved understanding of pharmacokinetic principles."

Discussion of the Findings

The evaluations of the quality of the learning object completed by faculty indicate that it will be a valuable resource for instruction. It is interesting to note that the content quality and the instructor/ student guides received the highest rating and reflects the amount of work that was done to select the appropriate strategies for the learning object. The high ratings for motivation reinforce the constructivist notion that activities which engage the learner are perceived as being more relevant and therefore more applicable for students. The ‘very good’ rating for reusability was somewhat surprising given the amount of modifications that were identified in order for it to be used in other contexts. The ‘low score’ by the one evaluator in the area of feedback/ adaptation may indicate that more direction is necessary for generating the graphs.

The ‘learning impact study lite’ served to reinforce the assumptions about the target audience, and the prior knowledge required by students in order to use the learning object. The results of the ‘usability of technology’ subscale (Agree and Strongly Agree 92.0%) reflects the prevalent role that technology plays in student learning. The 75.7% and 77.5% (Agree and Strongly Agree) responses for the Learning Value subscale and the Value Added subscale suggests that the learning object will help students to better understand the therapeutic principles of drug administration. Most significantly, this reinforces the design decisions about the scope and sequence of the learning object. This also supports the rationale for developing a learning object that could convey the impact that a number of variables have on the overall effect of the drug and the achievement of optimum therapeutic effect.

References

Belfer, K., Nesbit, J.C., Archambault, A., & Vargo, J. (2002) Learning object review instrument (LORI). Version 1.4. http://elera.matchbox.surrey.sfu.ca/eLera/Home