Introduction
Formative evaluation is
an essential component of the third phase of design-based research as the
learning solution is tested and refined. This evaluation involved the
intended facilitators of the e-learning course (including the project sponsor
and researcher) and learning consultant to examine the alpha version in order
to achieve these goals:
- Optimize the learning environment before course is released for pilot
- Conduct a sequential walkthrough of entire working version of course
- Identify potential issues with learning materials (e.g., activities, instructions)
- Achieve consensus on expectations for learning activities (e.g., acceptable/non-acceptable results)
- Begin to develop a reference guide for facilitators
- Identify significant risks and ways to control and mitigate them.
It was the last
goal in the list above that involved risk assessment that is described
here.
Risk assessment
Background on risk assessment and risk
management
Risk assessment and risk management are used in
almost every industry and profession to make data-supported, proactive
decisions on how to best use resources to prevent the occurrence of unwanted
events, and should they occur, to protect the assets of value in the
environment. Despite the usefulness of risk assessment in enabling potentially
problematic events to be articulated and then possibly accommodated, such
assessments are only rarely performed in planning e-learning environments.
Nevertheless, such risks do exist. While e-learning environments have inherent
(and easily predicted) risks related to data security, data loss and technology
failure, more subtle risks related to learning activities and assessment can
create critical obstacles for students engaged in e-learning. These risks
are compounded when e-learners in different countries and different cultures
must collaborate online.
Risk assessment is defined as the “overall
process of risk identification, risk analysis, and risk evaluation” (ISO, 2009). In
performing a risk assessment, one seeks answers to five basic
questions (Kaplan & Garrick, 1981):
1) What can go
wrong?
2) How bad can it
get?
3) How could it
happen?
4) How likely is it
to happen?
5) Should we try to
do something about this?
With answers to these questions, one can then
move into risk management where three other questions are asked (Haimes,
1991):
1) What can be done
to control, mitigate or prepare for this unwanted event?
2) What are the
best options given the circumstances?
3) What other risks
or issues might the selected option(s) create?
These questions are asked in a series of phases
using a variety of well-defined methods and tools to document the process and
results. Figure 1 shows a model of a typical risk assessment and risk
management process.
 |
| Figure 1. The typical process for risk assessment and risk management (Vesper, 2006) |
Risk
assessment can be performed using a variety of tools (such as those illustrated
in the right column of Figure 1). Some tools are very basic and may be informal,
for example, simply asking “what if…” questions. Other tools, like fault
tree analysis (FTA) and failure mode effects analysis (FMEA)
are highly structured and well-defined (Stamatis,2003; Vesely, Goldberg,
Roberts & Haasl, 1981). Certain tools are optimized to help identify
hazards – hazard analysis or hierarchical holographic modeling – while others
like hazard analysis and critical control points go
through the entire risk assessment and risk management process (Vesper,
2006).
There
is limited literature on risk assessment in relation to formative evaluation.
Lynch and Roecker (2007) recommended that risk assessment be used as part
of an evaluation and presented a simple form to collect data to be used in the
assessment. Similarly, Benson and Brack (2010), in their planning guide for
online learning and assessment, noted that an important administrative function
in planning online assessment was the completion of a risk assessment of: 1)
student support factors (such as access and equity issues), 2) technical issues
(such as access to hardware and software, bandwidth, etc.), 3) authentication
(such as cheating, collusion, plagiarism, etc.), and 4) consideration of the
instructor’s administrative skills (such as ability to use software, manage
online grading, copyright, etc.). However, no model or framework of risk
assessment appeared to exist that provided guidelines for the assessment of a
complex online authentic learning environment involving a community of
learners. In the next section, we describe the design and development of such a
framework.
Getting started
Before
starting a risk assessment, what is being assessed must be clearly defined.
This can be done by a written description, flowchart, or diagram (ICH, 2005).
For this project, the scope of the risk assessment include
the:
- e-learning application
- Technological infrastructure enabling the use of the
application
- All participants in the course (including the learners
and the facilitators/mentors)
One
other important but often overlooked element is clearly defining the “risk
question” – the question that the risk assessment is meant to answer (Vesper,
2006). This is consistent with one of Reeves and Hedberg’s (2003) key
reasons for doing a formative evaluation – answering questions that can be used
to make decisions about development.
Examples
of risk questions include:
1. What are the
IT/technology risks associated with this e-learning project?
2. What are the risks
related to the community of learners due to inappropriate communication?
3. What are all the risks
that could arise when using this e-learning program?
As
can be seen in these examples, risk questions can define the scope of the risk
assessment from very narrow (Risk question 2) to very wide (Risk question 3).
Often, the risk question drives the selection of the method the risk assessment
team selects. A preliminary risk assessment that asks, “What if...” could
be used with Risk questions 1 and 2; hierarchical holographic modeling (HHM)
and risk ranking and filtering are appropriate for identifying and assessing risks
in a large, complex system (Haimes, Kaplan &Lambert, 2002) such as
those that would be examined in answering Risk question 3.
Identifying hazards
Two
important definitions to distinguish between are hazard – the
source of harm – and risk – the combination of the likelihood of
the occurrence of the unwanted event resulting in the harm and the impact of
that harm (ICH, 2005). When starting a risk assessment, one first needs to
identify the hazards. There are different ways to identify hazards. A frequently
used method is to simply brainstorm what could go wrong. Other tools,
like hierarchical holographic modeling (Haimes, et al.,
2002) can be used to first create “success scenarios” from which risk
scenarios and specific risks can be identified.
In
this formative evaluation, the evaluation team first brainstormed what would be
necessary for a successful e-learning Pharmaceutical Cold Chain Management
Course (e-PCCMC). The team then identified actions, events, or situations
– the hazards – that could prevent or interfere with e-PCCMC. The list was then
condensed based on those hazards that were considered most relevant, and then
discussed further using a preliminary risk assessment tool.
Determining the risks
A
preliminary risk assessment (PRA) can be used early on in a project when
minimal information is available, or as a screening tool to identify risks that
need to be examined more critically using other tools, such as fault tree
analysis or failure mode effects analysis (Vesper, 2006). For the purposes of
this evaluation, the researcher felt that the PRA would provide an appropriate
level of detail.
For
each of the hazards, specific questions were asked to help determine the risk.
These included:
1. What are the potential
negative impacts to the learners and the desired course outcomes?
Answers to this question provided examples of the consequences, or harm should
the hazard be expressed.
2. What could cause this
unwanted event to occur? Here, the team identified how the hazard
could be expressed.
With
this information summarized using a matrix (see Figure 2), the team estimated the likelihood
that the hazard would be expressed resulting in the harm, using a scale of
low-medium-high (1-2-3) (Column 5). In a similar way, the impact was estimated, again
using a scale of low-medium-high (1-2-3) (Column 6). Multiplying these two numbers
resulted in a risk score – the higher the number the more risk being
present (Column 7).
The
last step of risk assessment is risk evaluation: deciding on the risks
that need to be reduced (Column 8). Generally, these are the high or medium risks that are
“treated” through control and mitigation. Other, low-level risks might be
addressed as well if the benefit outweighs the risk-reduction cost.
Reducing the risks through “treatment”
Risk
treatment (ISO, 2009) involves two key concepts: control and
mitigation. Control is aimed at preventing the unwanted event
from occurring in the first place; the focus is on reducing the likelihood by
targeting the root and contributing causes. Mitigation assumes the unwanted
event will occur but aims at protecting the “thing of value”
(CSA, 2002). For example, one cannot totally prevent a server crash at a
hosting site, but one can take protective measures should that happen. Whenever
possible, multiple risk treatment approaches should be taken that have a
“layering” of the control and mitigation actions. These are tied to the
different causes or mechanisms that were identified. These layers result in a
more robust solution should the hazard be expressed.
For
each of the risks that were identified, the team identified a risk treatment
plan. In some cases, it was providing information, for example,
recommending browsers that were tested (and what browsers are not
recommended). Another example of an identified risk was certain
governments not allowing access to a video website. A mitigation plan was
established to pre-make DVDs and send them by DHL courier to course
participants when requested by them. (This actually occurred – actually
occurred at the start of the pilot course. A participant could not access
the VIMEO or back-up sites, so the treatment plan – sending a pre-made DVD to
him via DHL – was executed.)
 |
| Figure 2. A section of a risk assessment performed using Preliminary Risk Assessment (PRA) worksheet |
Monitoring and review
The
identified risks were addressed through control and mitigation, however, team
members intended to review the assessment at the end of the pilot course to see
if the control and mitigation actions were effective and if the likelihood and
impact were correctly estimated. Additionally, monitoring was implemented as an
ongoing effort to determine if anything that changed that could affect the
assessment. Another aspect of monitoring is to identify any other risks
that were not previously identified.
A
formal review of the risk assessment and risk management plan will be performed
when the pilot course is completed. In terms of monitoring, the design
team realized two weeks into the course that there was going to be a seasonal
time change (from “standard time” to “daylight savings time”) occurring at two
different points during the course. To mitigate the impact, a notice was
sent to all participants alerting them to the change. This event will be
included in the listing of risks to be compiled for the next offering of the
course.
PRA
JIM MENTORS FEB2013 from EPELA on Vimeo.
This is an excerpt from a dissertation chapter
from James Vesper. Professors Thomas Reeves and Jan Herrington have contributed to it.
