Instruction > Testing & Evaluation

Testing & Evaluation

Assessment Types

Uses of Assessment  

Kirkpatrick's Levels of Assessment

In 1959 Donald Kirkpatrick developed a model of training evaluation. It is currently the most widely used evaluation approach. It is simple, flexible and complete using a four level approach. Recently, a fifth level emphasizing return on training investment (ROI)has been added by Jack Phillips:

Question Types: Quiz, Test, Interactive Practice

• Multiple-choice: choosing one answer from given alternatives (each response can be given a different score)
• Multiple response: choosing a number of answers from a list
• True/False: applied to a statement
• Yes/No: applied to a choice
• Lykert Scale: rating scale to survey preferences & opinions
• Fill-in-the-blank: text match can be used for simple text input
• Matching: selection/association, match items from two related lists
• Assertion/Reason: select correct reason for a particular assertion
• Multiple Hotspots: move labels to an appropriate place on an image
• Drag & Drop: manipulate graphics,by placing them into a particular sequence or arrangement
• Short answer: text match used for multi-line text box
• Essay: text match used for large multi-line text box
• Games: complex programmatic interactivity
• Simulation: interactive practice of a task
• Custom question types: using audio-video enhancements, problem-solving, visualization or modeling   

Benefits of Web-based Testing

• Time and cost savings in test administration & automated scoring  
• Greater score precision
• Maximize student engagement: minimize student frustration in taking test that are too difficult or boredom in taking test that are too easy
• Improved test security
• New kinds of questions by using multimedia, simulations , visualizations, and other resources to assess active learning

Bloom's Taxonomy: Cognitive Domain   

An introduction to Bloom's Domains of Learning was presented in an earlier section "Types of Instruction by Domains of Learning: Knowledge, Attitude & Skills". The following is organized to emphasize the importance of this model of the Cognitive Domain for measuring performance and creating effective tests.

The major categories in the cognitive domain of educational objectives, edited by Benjamin Bloom (1956), are the following.

Knowledge: Remember and Recall; Observe and recognize information, Demonstrate knowledge of dates, events, places; Demonstrate knowledge of major ideas; Demonstrate mastery of basic subject matter.

Comprehension: Translate and Paraphrase; Interpret and Extrapolate information; Use information, methods, concepts theories in new situations; Solve problems; Use required skills or knowledge; Ability to grasp meaning; Estimating future trends (predicting consequences or effects)

Application: Transfer knowledge to new settings; Use of Generalizations in Specific instances; Ability to use learned material in new and concrete situations (rules, principles, laws, theories)

Analysis: Differentiate component parts; Determine Relationships; Breakdown ideas or structure; Identification of parts and organizational principles; Determine arrangement, logic and semantics

Synthesis: Weave components into a whole; Create New Relationships; Use old ideas to create new ones; Generalize from given facts; Relate knowledge from several areas; Predict; Draw conclusions; The ability to put parts together for form a new whole

Evaluation: Judge the value of information; Exercise of learned judgment; Evaluate based on given criteria; The ability to judge the value of material (i.e., statement, novel, poem, research report) for a given purpose

Bloom, B.S. (Ed.). (1956). Taxonomy of Educational Objectives: Classification of Educational Goals. Handbook 1: Cognitive Domain. New York: Longman, Green & Co.
 

Test Development: Question Stems and Verbs   

LORI -- Learning Object Review Instrument

The Learning Object Review Instrument (LORI) is used to evaluate the quality of E-Learning resources. LORI is an online form that consisting of rubrics, rating scales, and comment fields. The current version of LORI available from eLera is version 1.5. Here is a summary of the criteria used for evaluation:

E-Learning Program Evaluation: Brandon-Hall

The following evaluation criteria was developed by Brandon Hall and Arjun Reddy in 1988 (MM&ITNL,Brandon-Hall) for evaluation of E-Learning programs:

Evaluation Categories	Evaluation Criteria
1. Content	Right amount and quality of information   Content meets defined objectives  Content organization
2. Instructional     Design	Course objectives clearly defined & tangible   Chunking of information  Job aids / performance support
3. Interactivity	User engaged through the opportunity for their input   Amount of interactivity   Quality of interactivity
4. Navigation	Users can determine their own path through the program   An exit option always available   A course map is always accessible   Appropriate use of icons and/or clear labels so that users don't have to read excessively to determine program options   Clarity of directions   Adequacy of navigation controls   Branching to other topics doesn't create a sense of being 'lost'
5. Motivation	Program engages the user through novelty, humor, game elements, testing, adventure, unique content, surprise elements, etc.   The course follows a metaphor (golf, Doom, etc.)   Metaphor effectively used to help the learning process   Progress indicator
6. Use of Media	Program effectively and appropriately employs: video, audio (voice, music, sound effects), animation, graphics, special visual effects
7. Evaluation	Mastery of a section required before proceeding to later sections   Section quizzes used   Final exam   Quality of the testing modules   Testing is relevant to the real world performance objectives   Appropriateness and timeliness of feedback
8. Aesthetics	Program attractive and appealing to eye and ear   Design of the interface is simple, uncluttered, symmetry of objects such as headings, menu bar, etc. Screens do not look busy.
9. Record     Keeping	Student performance data recorded, such as time to complete, question analyses, and final score   Data forwarded to course manager automatically
10. Tone	Program designed for intended audience   Program avoids being condescending, trite, etc.

In 2002 the following criteria was used by Brandon Hall for evaluation of E-Learning programs:

Evaluation Categories	Evaluation Criteria
1. Business problem & results	Matching the e-learning solution to the business/ performance problem at hand   Achieving intended results.
2. Instructional design & integrity	Structuring, relevance and quality of content;   Focus on real-world competencies;  Selecting the right strategies for the content and context.
3. Evaluation & assessment	Applying imagination and rigor to the design and implementation of evaluation or assessment
4. Interactivity	Using creativity and expert design practices to achieve instructionally powerful interactions of all kinds
5. Usability & Interface	Creating an effective, easy-to-use interface
6. Motivation & aesthetics	Motivating learners to follow and successfully complete the training   Hitting the right tone and aesthetic notes
7. Media & technology	Smart selection and application of media, development tools and delivery technologies
8. Money & time	Achieving excellence under constrained budgets and time lines

Five Star Evaluation: M. David Merrill    

The following principles have been proposed by M.David Merrill as a way of getting at the most important criteria for evaluating E-Learning effectiveness. A more detailed discussion has been presented in the section "5 Star Instruction."

5 Star Evaluation	Merrill's Evaluation Criteria
1. Problem Principle	Learning is facilitated when learners are engaged in solving real-world problems  The learner is engaged at the problem or task level, not just the operation or action level  The learner solves a progression of problems. This promotes skill development, and meaningful feedback and reinforcement to occur.  The learner is guided to an explicit comparison of problems.
2. Activation Principle	Learning is facilitated when existing knowledge is activated as a foundation for new knowledge  The learner is directed to recall, relate, describe, or apply knowledge from relevant past experience that can be used as a foundation for the new knowledge.  The learner is provided with relevant experience that can be used as a foundation for the new knowledge.
3. Demonstration Principle	Learning is facilitated when new knowledge is demonstrated to the learner  The Learner is shown rather than merely told.   The demonstration is consistent with the learning goal.   The learner is shown multiple representations.   The learner is directed to explicitly compare alternative represenations.   The use of media play a relevant instructional role.
4. Application Principle	Learning is facilitated when new knowledge is applied by the learner (e.g., guided and unguided practice)  The learner is required to use his or her new knowledge to solve problems.  The problem-solving activity is consistent with the learning goal.  The learner is shown how to detect and correct errors.   The learner is guided in problem-solving by appropriate coaching that is gradually withdrawn.
5. Integration Principle	Learning is facilitated when new knowledge is integrated into the learner's world   The learner is required to demonstrate his or her new knowledge or skill.   The learner can reflect-on, discuss, and defend his or her new knowledge or skill.   The learner can create, invent, and explore new and personal ways to use his or her new knowledge or skill

Conscious Competence Learning Matrix  

The following model was developed to describe some key observations about learning and performance — and the assessment of performance . The learner of a new skill begins at stage 1 - 'unconscious incompetence', and ends at stage 4 - 'unconscious competence', having passed through stage 2 - 'conscious incompetence' and - 3 'conscious competence'.

Trainers often make the mistake of assuming that a learner is at stage 2, and focus their effort toward achieving stage 3, when in fact the learner is still at stage 1. This is a a fundamental reason for training failure — because the learner simply has not recognized the need for new learning. Until the learner has achieved awareness of a weakness or a training need ('conscious incompetence'), the learner has no interest, attention or motivation for the learning process. Learners only respond to training or teaching when they are aware of their own need for it, and the personal benefits they will derive from it.

This model would be incomplete if it did not acknowledge the idea of a 5th level which has sometimes been called "reflective competence" or even "enlightened competence". The model illustrates how skills become so integrated that they become "unconscious" and instinctual. But if we stopped there, it would give the impression that this "unconsciousness" is the highest stage of learning — such as the artist, dancer, craftsman who practices their skill at the highest level but cannot articulate it or teach it to others. Of course, the level beyond this is the coach, the expert teacher, the mentor, the master craftsman, who can demonstrate a skill in practice and also articulate the fine details of the skill, art or craft.

References   

Bloom, B.S. (Ed.). (1956). Taxonomy of Educational Objectives: Classification of Educational Goals. Handbook 1: Cognitive Domain. New York: Longman, Green & Co.

Kirkpatrick, Donald, (1994) Evaluating Training Programs: The Four Levels. San Francisco, CA: Berrett-Koehler Publishers, inc., 2nd Ed, 1998

Merrill, M. David, Five Star Instruction (2003); and First Principles of Instruction (2002.)

Nantel, Richard and staff of Brandon Hall Research (2005). Testing Building Tools: A Comparison of 24 Products for Authoring Online Tests, Assessments and Evaluations. Brandon Hall Research, Sunnyvale, CA.
http://www.brandonhall.com/public/publications/testbuildingtools/index.htm

Instruction Overview

Testing & Evaluation