Using Risk Assessment as Part of an Early Formative Evaluation

Introduction

Formative evaluation is an essential component of the third phase of design-based research as the learning solution is tested and refined.  This evaluation involved the intended facilitators of the e-learning course (including the project sponsor and researcher) and learning consultant to examine the alpha version in order to achieve these goals: 

• Optimize the learning environment before course is released for pilot
• Conduct a sequential walkthrough of entire working version of course
• Identify potential issues with learning materials (e.g., activities, instructions)
• Achieve consensus on expectations for learning activities (e.g., acceptable/non-acceptable results)
• Begin to develop a reference guide for facilitators
• Identify significant risks and ways to control and mitigate them. 

It was the last goal in the list above that involved risk assessment that is described here. 

Risk assessment

Background on risk assessment and risk management

Risk assessment and risk management are used in almost every industry and profession to make data-supported, proactive decisions on how to best use resources to prevent the occurrence of unwanted events, and should they occur, to protect the assets of value in the environment. Despite the usefulness of risk assessment in enabling potentially problematic events to be articulated and then possibly accommodated, such assessments are only rarely performed in planning e-learning environments. Nevertheless, such risks do exist. While e-learning environments have inherent (and easily predicted) risks related to data security, data loss and technology failure, more subtle risks related to learning activities and assessment can create critical obstacles for students engaged in e-learning.  These risks are compounded when e-learners in different countries and different cultures must collaborate online. 

Risk assessment is defined as the “overall process of risk identification, risk analysis, and risk evaluation” (ISO, 2009). In performing a risk assessment, one seeks answers to five basic questions (Kaplan & Garrick, 1981):

1)     What can go wrong?

2)     How bad can it get?

3)     How could it happen?

4)     How likely is it to happen?

5)     Should we try to do something about this?

With answers to these questions, one can then move into risk management where three other questions are asked (Haimes, 1991):

1)     What can be done to control, mitigate or prepare for this unwanted event?

2)     What are the best options given the circumstances?

3)     What other risks or issues might the selected option(s) create?

These questions are asked in a series of phases using a variety of well-defined methods and tools to document the process and results.  Figure 1 shows a model of a typical risk assessment and risk management process.

Figure 1. The typical process for risk assessment and risk management (Vesper, 2006)

Risk assessment can be performed using a variety of tools (such as those illustrated in the right column of Figure 1). Some tools are very basic and may be informal, for example, simply asking “what if…” questions. Other tools, like fault tree analysis (FTA) and failure mode effects analysis (FMEA) are highly structured and well-defined (Stamatis,2003; Vesely, Goldberg, Roberts & Haasl, 1981). Certain tools are optimized to help identify hazards – hazard analysis or hierarchical holographic modeling – while others like hazard analysis and critical control points go through the entire risk assessment and risk management process (Vesper, 2006). 

There is limited literature on risk assessment in relation to formative evaluation. Lynch and Roecker (2007) recommended that risk assessment be used as part of an evaluation and presented a simple form to collect data to be used in the assessment. Similarly, Benson and Brack (2010), in their planning guide for online learning and assessment, noted that an important administrative function in planning online assessment was the completion of a risk assessment of: 1) student support factors (such as access and equity issues), 2) technical issues (such as access to hardware and software, bandwidth, etc.), 3) authentication (such as cheating, collusion, plagiarism, etc.), and 4) consideration of the instructor’s administrative skills (such as ability to use software, manage online grading, copyright, etc.). However, no model or framework of risk assessment appeared to exist that provided guidelines for the assessment of a complex online authentic learning environment involving a community of learners. In the next section, we describe the design and development of such a framework.

Getting started

Before starting a risk assessment, what is being assessed must be clearly defined. This can be done by a written description, flowchart, or diagram (ICH, 2005).  For this project, the scope of the risk assessment include the:  

• e-learning application
• Technological infrastructure enabling the use of the application
• All participants in the course (including the learners and the facilitators/mentors)

One other important but often overlooked element is clearly defining the “risk question” – the question that the risk assessment is meant to answer (Vesper, 2006). This is consistent with one of Reeves and Hedberg’s (2003) key reasons for doing a formative evaluation – answering questions that can be used to make decisions about development. 

Examples of risk questions include:

1.    What are the IT/technology risks associated with this e-learning project?

2.    What are the risks related to the community of learners due to inappropriate communication?

3.    What are all the risks that could arise when using this e-learning program?

As can be seen in these examples, risk questions can define the scope of the risk assessment from very narrow (Risk question 2) to very wide (Risk question 3). Often, the risk question drives the selection of the method the risk assessment team selects.  A preliminary risk assessment that asks, “What if...” could be used with Risk questions 1 and 2; hierarchical holographic modeling (HHM) and risk ranking and filtering are appropriate for identifying and assessing risks in a large, complex system (Haimes, Kaplan &Lambert, 2002) such as those that would be examined in answering Risk question 3. 

Identifying hazards

Two important definitions to distinguish between are hazard – the source of harm – and risk – the combination of the likelihood of the occurrence of the unwanted event resulting in the harm and the impact of that harm (ICH, 2005). When starting a risk assessment, one first needs to identify the hazards. There are different ways to identify hazards. A frequently used method is to simply brainstorm what could go wrong. Other tools, like hierarchical holographic modeling (Haimes, et al., 2002) can be used to first create “success scenarios” from which risk scenarios and specific risks can be identified. 

In this formative evaluation, the evaluation team first brainstormed what would be necessary for a successful e-learning Pharmaceutical Cold Chain Management Course (e-PCCMC).  The team then identified actions, events, or situations – the hazards – that could prevent or interfere with e-PCCMC. The list was then condensed based on those hazards that were considered most relevant, and then discussed further using a preliminary risk assessment tool. 

Determining the risks

A preliminary risk assessment (PRA) can be used early on in a project when minimal information is available, or as a screening tool to identify risks that need to be examined more critically using other tools, such as fault tree analysis or failure mode effects analysis (Vesper, 2006). For the purposes of this evaluation, the researcher felt that the PRA would provide an appropriate level of detail. 

For each of the hazards, specific questions were asked to help determine the risk. These included:

1.   What are the potential negative impacts to the learners and the desired course outcomes?  Answers to this question provided examples of the consequences, or harm should the hazard be expressed.

2.   What could cause this unwanted event to occur?  Here, the team identified how the hazard could be expressed.

With this information summarized using a matrix (see Figure 2), the team estimated the likelihood that the hazard would be expressed resulting in the harm, using a scale of low-medium-high (1-2-3) (Column 5). In a similar way, the impact was estimated, again using a scale of low-medium-high (1-2-3) (Column 6). Multiplying these two numbers resulted in a risk score – the higher the number the more risk being present (Column 7). 

The last step of risk assessment is risk evaluation:  deciding on the risks that need to be reduced (Column 8). Generally, these are the high or medium risks that are “treated” through control and mitigation.  Other, low-level risks might be addressed as well if the benefit outweighs the risk-reduction cost. 

Reducing the risks through “treatment”

Risk treatment (ISO, 2009) involves two key concepts:  control and mitigation. Control is aimed at preventing the unwanted event from occurring in the first place; the focus is on reducing the likelihood by targeting the root and contributing causes. Mitigation assumes the unwanted event will occur but aims at protecting the “thing of value” (CSA, 2002). For example, one cannot totally prevent a server crash at a hosting site, but one can take protective measures should that happen. Whenever possible, multiple risk treatment approaches should be taken that have a “layering” of the control and mitigation actions. These are tied to the different causes or mechanisms that were identified. These layers result in a more robust solution should the hazard be expressed.

For each of the risks that were identified, the team identified a risk treatment plan.  In some cases, it was providing information, for example, recommending browsers that were tested (and what browsers are not recommended).  Another example of an identified risk was certain governments not allowing access to a video website.  A mitigation plan was established to pre-make DVDs and send them by DHL courier to course participants when requested by them.  (This actually occurred – actually occurred at the start of the pilot course.  A participant could not access the VIMEO or back-up sites, so the treatment plan – sending a pre-made DVD to him via DHL – was executed.) 

Figure 2. A section of a risk assessment performed using Preliminary Risk Assessment (PRA) worksheet

Monitoring and review

The identified risks were addressed through control and mitigation, however, team members intended to review the assessment at the end of the pilot course to see if the control and mitigation actions were effective and if the likelihood and impact were correctly estimated. Additionally, monitoring was implemented as an ongoing effort to determine if anything that changed that could affect the assessment.  Another aspect of monitoring is to identify any other risks that were not previously identified. 

A formal review of the risk assessment and risk management plan will be performed when the pilot course is completed.  In terms of monitoring, the design team realized two weeks into the course that there was going to be a seasonal time change (from “standard time” to “daylight savings time”) occurring at two different points during the course.  To mitigate the impact, a notice was sent to all participants alerting them to the change.  This event will be included in the listing of risks to be compiled for the next offering of the course.

PRA JIM MENTORS FEB2013 from EPELA on Vimeo.

This is an excerpt from a dissertation chapter from James Vesper. Professors Thomas Reeves and Jan Herrington have contributed to it.