2 Rovai, “A Practical Framework for Evaluating Online Distance Education Program
2 Rovai, “A Practical Framework for Evaluating Online Distance Education Programs”
Reference
Background
Rovai (2003) argued in this research article that, in order for distance education to be as successful as possible, evaluations must be done and they must be thorough. He proposed a framework following a review of the literature surrounding online education evaluation and provides specific areas where the foci should be for distance education evaluation. Further, Rovai (2003) offered questions that cover four areas of evaluation for stakeholders and the methods needed to answer the questions. Those four areas of evaluation highlighted were:
• Input
• Output
• Process
• Impact
The framework and suggestion the author made relied heavily on open-systems theory, which supports a flexible set of processes to match what would be most appropriate for the program (Rovai, 2003). Evaluations concluded with the analysis of gathered information. This process was modeled in Figure 1. All opinions were considered for inclusion in the final report and the final comments included the perspectives that best represented the interests of the audience and stakeholders. Finally, programs were judged based on their norms (program standards and expectations) or benchmarks were put in place for the final judgment to take place (Rovai, 2003). It is significant to note that benchmarks need to be determined prior to the evaluation process beginning so as to avoid end-of-evaluation bias or changing standards to reflect the evaluation, meaning norms need to be a significant part of the initial conversation with stakeholders (Rovai, 2003).
Key points
- While there are a variety of evaluation categorizations to choose from (formative, summative, CIPP, etc), the author argued that the systems model approach was the most effective for distance education given its ability to evaluate multifaceted program elements (Rovai, 2003).
- Program evaluations should include:
- Student performance
- Program cost and effectiveness
- Course design and instruction
- Teacher and student satisfaction
- Quality of technology and support services
- Specific questions and methodologies should be employed to meet the needs of the stakeholders. Qualitative and/or quantitative methodology should be used to answer the evaluation questions. Four evaluation types were provided:
- Process
- Impact
- Output
- Input
- Within the specific evaluation types, there are specific orientation strategies that reflect stakeholder involvement to consider. These strategies can either be used independently or combined with one another to build the most accurate evaluation:
- Objectives-oriented—Determining whether outcomes and program goals were met
- Management-oriented—Determining funding (reallocation, especially)
- Consumer-oriented—Developing information on the product for the consumer
- Expertise-oriented—Professional judges an educational program (heavily used for accreditation purposes)
- Adversary-oriented—I