On 22 February 2015, we lost Andrew Garnett...  A dear friend, mentor, and an irreplaceable brain… A huge loss to the world of immunization.

Andrew Garnett, during the EPELA e-Pharmaceutical Cold Chain Management expert video shooting, 15 March 2011, London (photo: U. Kartoglu)

I met Andrew in 2001 during the revision of international packaging and shipping guidelines. From that day on, we became close friends and have worked on numerous projects. It is with Andrew that we brought to life the Effective Vaccine Store Management (EVSM) initiative and Performance,  Quality and Safety (PQS) project.

Andrew was the number one choice as an author on cold chain management guidelines. He was an immaculate editor for many technical series we worked on together.

As an architect, Andrew was THE master of mind-maps. He fascinated me with his ability to bring about a clear mind-map structure to any complex concept.  

Andrew was one of the four mentors teaching in EPELA e-Pharmaceutical Cold Chain management course. His insightful reviews and feedback to participants were deeply appreciated and will always be remembered.

I deeply admired you, Andrew, for your elegance, your subtle artistic taste, your scientific rigor, your passion for life…

I shall miss you dearly, Andrew.

Umit Kartoglu 

New expert videos on VVM based vaccine management

EPELA releases the most recently produced expert videos on VVM based vaccine management. These videos are specifically produced for the new EPELA authentic e-learning course on vaccine vial monitor based vaccine management. The new course will be offered during 27 January - 28 March 2014.

Both WHO and EPELA encourage others to widely use, share the videos and embed them in other web sites. The following videos are now available publicly:

A year in the life of a vaccine by Kevin O'Donnell

Kevin O’Donnell reviews the role of VVM and how it can be the answer to increase access and ensure quality of vaccine was not compromised due to unacceptable heat exposure.

Cold chain challenges everywhere by Simona Zipursky

Simona Zipursky reviews the cold chain challenges to demonstrate that problems are both in developing and industrialized countries and questions whether VVM should also be the answer for both.

How does a VVM work by Denis Maire

Denis Maire summarizes the technical characteristics of VVMs and explains how they work.

Interpretation of VVM in relation to other temperature monitoring devices by Umit Kartoglu

Umit Kartoglu reviews temperature monitoring devices used in a typical vaccine cold chain and analyses how the readings relate to each other when there are more than one device at a particular point. This analysis is done from the VVM perspective.

Using VVM as a stock management tool by Umit Kartoglu

Umit Kartoglu reviews the requirements for product arrival, storage and dispatch and analyses the role of VVM in effective stock management for each step. Special emphasis is given to the relation of VVM and expiry date in illustrating how VVM over-rules earliest expiry first out principle.

Vaccines beyond the cold chain by Simona Zipursky

Simona Zipursky reviews the studies on taking vaccines beyond the cold chain all published in peer-review journals and comments on how VVMs could be instrumental in these operations.

VVMs getting smarter by Umit Kartoglu

Umit Kartoglu reviews the recent changes in integrity and location of VVMs and the new message VVM is giving whether a vial containing multi-dose vaccine following opening can be kept for a subsequent session or not.

VVM use at the most periphery by Serge Ganivet

Serge Ganivet reviews the VVM use at the most periphery through different examples and brings new perspectives on how to make best decisions based on the expiry and VVM readings.

EPELA launches the new authentic e-learning course on VVM based vaccine management

EPELA launches its new authentic e-learning course on VVM based vaccine management. 

The 9-week course takes participants through a sophisticated vaccine chain to make the best use of VVM through ensuring that the vaccines have not been damaged by heat, reducing vaccine wastage, pinpointing cold chain problems, managing stocks, facilitating immunization outreach and increasing access and improving vaccination coverage, preventing inadvertent freezing, and providing hints whether an opened multi-dose vaccine vial may be used in a subsequent immunization session. 

Tools are offered to work in groups, evaluation is done through self, peer and expert reviews. Nobody lectures any participants in this course, but we make expert videos and critical documents available to participants through video and document libraries. 

The course will be mentored by Umit Kartoglu, Julie Milstien, Ticky Raubenheimer and Denis Maire. 

The new course will run as beta-course for the first time between 27 January and 28 March 2014. Regular courses are scheduled to take place during the following periods:

21 April - 20 June 2014

6 October - 5 December 2014

For details of the course please check at http://epela.net/epela_web/evvm.html  

New graduates of the e-Pharmaceutical cold chain management, 30 September - 20 December 2013

Congratulations to our new graduates...

Thomas Reeves, our senior advisor, receives two distinguished awards at the 2013 AECT International Convention

Our dear friend, e-learning guru and mentor, and senior advisor to our e-learning courses Thomas Reeves has received two distinguished awards at the 2013 AECT International Convention (Association for Educational Communications and Technology) conference in Anaheim, CA last week. We are very proud with you Tom!

OUTSTANDING BOOK AWARD


Tom was awarded with OUTSTANDING BOOK AWARD together with Susan McKinney for their “Conducting educational design research” book published by Routledge (24 February 2012). Educational design research blends scientific investigation with systematic development and implementation of solutions to educational problems. Empirical investigation is conducted in real learning settings-not laboratories-to craft usable and effective solutions. At the same time, the research is carefully structured to produce theoretical understanding that can serve the work of others. To support graduate students as well as experienced researchers who are new to this approach, Conducting Educational Design Research integrates multiple perspectives of educational design research throughout this three-part book. Part one clarifies the educational design research origins, approach and outcomes. It also presents a generic model portraying the overall process. Part two discusses the constituent elements of the model in detail, these are: analysis and exploration; design and construction; evaluation and reflection; and implementation and spread. Part three offers recommendations for proposing, reporting and advancing educational design research. Through related readings and richly varied examples, Conducting Educational Design Research offers clear and well-documented guidance on how to conceptualize and conduct this stimulating form of inquiry. For decades, policies for educational research worldwide have swung back and forth between demanding rigor above all other concerns, and increasing emphasis on impact. These two qualities need not be mutually exclusive. The book supports readers in grasping and realizing the potential of educational design research. It demonstrates how rigorous and relevant investigation can yield both theoretical understanding and solutions to urgent educational problems.

We feel very privileged to have Tom on board with us in our learning journey and particularly in our e-learning initiative. It is also wonderful that Jim (Vesper) has applied this approach to our e-pharmaceutical cold chain management course.

David H. Jonassen Excellence in Research Award

Tom was also awarded with the “David H. Jonassen Excellence in Research Award”, given to individuals who have demonstrated a long-term record of excellence in research to the field of instructional design and technology. The award was established to remember the significant scholarly contributions made to the field of instructional design and technology by Dr. David H. Jonassen.

Tom also delivered a keynote speech at the conference titled “Educational Design Research: If Not Now, When?”.

I met Tom at the 2008 PDA Biennial training conference, 19-21 May 2008 in New Orleans, Louisiana, USA (also met Jim there). Since then I work together with these two beautiful individuals. Together with Tom (and Jim) we facilitate Designing courses for learning and Facilitation skills courses for WHO.

I (Umit) and Tom at the November 2009 “Designing

courses for learning" course, Antalya, Turkey

Tom is a Professor Emeritus of Learning, Design, and Technology, Department of Educational Psychology and Instructional Technology, College of Education, The University of Georgia. He earned hisPh.D. at Syracuse University. Tom was a Fulbright Lecturer in Peru and have been an invited speaker in the USA and 30 other countries.

From 1997-2000, Tom was the editor of the Journal of Interactive Learning Research. In 2003, he received the Fellowship Award from the Association for the Advancement of Computing in Education (AACE), and in 2010 he was made a Fellow of the Australasian Society for Computers in Learning in Tertiary Education (ASCILITE). 

Congratulations Tom!.. UMIT

Formative evaluation of the e-learning course "Pharmaceutical Cold Chain Management"

“Piloting” a course is a well-accepted way to tryout a learning solution with participants in order to make improvements and decide if the learning solution is ready to be launched with a larger, less “controlled” group of learners.  During a pilot, much more data is collected and comprehensively analyzed for formative purposes (i.e., making improvements) than is done during the first full implementation of the course.

The EPELA Pharmaceutical Cold Chain Management e-learning course went through its pilot release starting in March and concluding in early June 2013.  Fifteen people who had previously participated in the actual Pharmaceutical Cold Chain Management on Wheels courses between 2009 and 2012 agreed to participate in the pilot.  Because of work-related and personal reasons, four people withdrew from the course before it ended in June.

Integrated into the course was an online diary where each participant could reflect on what they learned; ideas, concepts and actions that they could take back and apply; and comments (e.g., observations, suggestions) related to that task.  Additionally, a link to an online questionnaire (using the application SurveyMonkey) for each task was provided in order to collect information pertinent to the formative evaluation of the pilot course.  At the end of the course, an hour-long teleconference was arranged with all mentors and participants talking about the course and what it meant to them.

The diaries, questionnaires, emails, and a final group evaluation teleconference were all carefully analyzed.  Participants and mentors made a number of positive comments about what they liked in the course and also what they thought could be improved.  From this data, a number of recommendations were given to the course director/sponsor, Dr. Umit Kartoglu by Jim Vesper, who was both a mentor in the pilot course and the primary evaluation researcher on this project.

In September 2013, several weeks before the start of the first full implementation of the course, Dr. Kartoglu met with Jim Vesper to clarify and implement the recommendations.

Most all of the recommendations could be made by changing instructions, adding more details to activities, and creating newer and better ways for participants and mentors to work together.

Table 1 shows some of the recommendations and actions that were taken to implement them.

Table 1.  Examples of improvements made to e-PCCM course based on recommendations from formative evaluation of pilot course (Mar-Jun 2013)

Recommendations	Actions	Before	After
1.    Review all learning objectives to assure that objectives are aligned to module content and tasks.	All learning objectives were reviewed with many revised to match the instructional intent of the module.  Some of the original objectives were either not covered in the completed modules or did not accurately reflect what was covered in the module.	1.    Identify the major operational components in a pharmaceutical cold chain 2.     Identify deficiencies in given situations using the “good distribution practice” (GDP) guidelines. 3.     For a given a mode of transportation, identify hazards, and assess and identify methods to control the risks to pharma, biopharma, and vaccine products that are consistent with GDP. 4.     For a given a situation, select the appropriate methods and materials for packaging and shipping cold chain products to minimize risk. 5.    For a given a situation, discuss specific requirements to be included in a quality agreement.	1.   Given a situation, propose recommendations to improve compliance with “good distribution practice” (GDP) guidelines. 2.   Given a nonconformance in the transport of pharmaceutical product, analyze data to identify the cause, potential impact to the product, and formulate preventive measures. 3.    Given a list of elements that could be in a quality agreement, justify five elements you consider to be most beneficial. 4.    Given an example of an operational component in a pharmaceutical cold chain, differentiate the practices as to whether or not they reduce risks.
2.     Review all learning objectives in light of activities/tasks that the participants perform and revise as necessary so that the objectives encompass higher levels of cognitive skills.	All learning objectives and activities/tasks were reviewed and verbs were selected to more clearly describe the cognitive skill that the learner was to achieve.	1.    For a given situation, select the appropriate methods and materials needed to monitor temperature and/or humidity for cold chain products and to obtain necessary data for making decisions.	1.   Given a stock situation with different vaccines, various expiry periods and batches and VVM status, decide which products to be dispatched against a requisition order. 2.    Create a decision tree for dispatch of vaccines involving all relevant factors.
3.     Develop a different, more complex activity related to risk assessment and risk management.	A new activity was created for the Hacettepe Hospitals visit.	Initial activity was to review a video and identify the controls in place and the risks that the controls were to reduce.	New activity was to perform a risk assessment on drug storage refrigerators in an isolated area that had only temperature read-outs and local alarms.

Conducting this type of formative evaluation takes time and effort from participants, course mentors, and researchers.  As can be seen from what was done with the EPELA e-PCCM course, formative evaluation can take a learning solution that was very good and make it even better!

e-Pharmaceutical cold chain management beta-course graduates with mentors and designers

Using Risk Assessment as Part of an Early Formative Evaluation

Introduction

Formative evaluation is an essential component of the third phase of design-based research as the learning solution is tested and refined.  This evaluation involved the intended facilitators of the e-learning course (including the project sponsor and researcher) and learning consultant to examine the alpha version in order to achieve these goals: 

• Optimize the learning environment before course is released for pilot
• Conduct a sequential walkthrough of entire working version of course
• Identify potential issues with learning materials (e.g., activities, instructions)
• Achieve consensus on expectations for learning activities (e.g., acceptable/non-acceptable results)
• Begin to develop a reference guide for facilitators
• Identify significant risks and ways to control and mitigate them. 

It was the last goal in the list above that involved risk assessment that is described here. 

Risk assessment

Background on risk assessment and risk management

Risk assessment and risk management are used in almost every industry and profession to make data-supported, proactive decisions on how to best use resources to prevent the occurrence of unwanted events, and should they occur, to protect the assets of value in the environment. Despite the usefulness of risk assessment in enabling potentially problematic events to be articulated and then possibly accommodated, such assessments are only rarely performed in planning e-learning environments. Nevertheless, such risks do exist. While e-learning environments have inherent (and easily predicted) risks related to data security, data loss and technology failure, more subtle risks related to learning activities and assessment can create critical obstacles for students engaged in e-learning.  These risks are compounded when e-learners in different countries and different cultures must collaborate online. 

Risk assessment is defined as the “overall process of risk identification, risk analysis, and risk evaluation” (ISO, 2009). In performing a risk assessment, one seeks answers to five basic questions (Kaplan & Garrick, 1981):

1)     What can go wrong?

2)     How bad can it get?

3)     How could it happen?

4)     How likely is it to happen?

5)     Should we try to do something about this?

With answers to these questions, one can then move into risk management where three other questions are asked (Haimes, 1991):

1)     What can be done to control, mitigate or prepare for this unwanted event?

2)     What are the best options given the circumstances?

3)     What other risks or issues might the selected option(s) create?

These questions are asked in a series of phases using a variety of well-defined methods and tools to document the process and results.  Figure 1 shows a model of a typical risk assessment and risk management process.

Figure 1. The typical process for risk assessment and risk management (Vesper, 2006)

Risk assessment can be performed using a variety of tools (such as those illustrated in the right column of Figure 1). Some tools are very basic and may be informal, for example, simply asking “what if…” questions. Other tools, like fault tree analysis (FTA) and failure mode effects analysis (FMEA) are highly structured and well-defined (Stamatis,2003; Vesely, Goldberg, Roberts & Haasl, 1981). Certain tools are optimized to help identify hazards – hazard analysis or hierarchical holographic modeling – while others like hazard analysis and critical control points go through the entire risk assessment and risk management process (Vesper, 2006). 

There is limited literature on risk assessment in relation to formative evaluation. Lynch and Roecker (2007) recommended that risk assessment be used as part of an evaluation and presented a simple form to collect data to be used in the assessment. Similarly, Benson and Brack (2010), in their planning guide for online learning and assessment, noted that an important administrative function in planning online assessment was the completion of a risk assessment of: 1) student support factors (such as access and equity issues), 2) technical issues (such as access to hardware and software, bandwidth, etc.), 3) authentication (such as cheating, collusion, plagiarism, etc.), and 4) consideration of the instructor’s administrative skills (such as ability to use software, manage online grading, copyright, etc.). However, no model or framework of risk assessment appeared to exist that provided guidelines for the assessment of a complex online authentic learning environment involving a community of learners. In the next section, we describe the design and development of such a framework.

Getting started

Before starting a risk assessment, what is being assessed must be clearly defined. This can be done by a written description, flowchart, or diagram (ICH, 2005).  For this project, the scope of the risk assessment include the:  

• e-learning application
• Technological infrastructure enabling the use of the application
• All participants in the course (including the learners and the facilitators/mentors)

One other important but often overlooked element is clearly defining the “risk question” – the question that the risk assessment is meant to answer (Vesper, 2006). This is consistent with one of Reeves and Hedberg’s (2003) key reasons for doing a formative evaluation – answering questions that can be used to make decisions about development. 

Examples of risk questions include:

1.    What are the IT/technology risks associated with this e-learning project?

2.    What are the risks related to the community of learners due to inappropriate communication?

3.    What are all the risks that could arise when using this e-learning program?

As can be seen in these examples, risk questions can define the scope of the risk assessment from very narrow (Risk question 2) to very wide (Risk question 3). Often, the risk question drives the selection of the method the risk assessment team selects.  A preliminary risk assessment that asks, “What if...” could be used with Risk questions 1 and 2; hierarchical holographic modeling (HHM) and risk ranking and filtering are appropriate for identifying and assessing risks in a large, complex system (Haimes, Kaplan &Lambert, 2002) such as those that would be examined in answering Risk question 3. 

Identifying hazards

Two important definitions to distinguish between are hazard – the source of harm – and risk – the combination of the likelihood of the occurrence of the unwanted event resulting in the harm and the impact of that harm (ICH, 2005). When starting a risk assessment, one first needs to identify the hazards. There are different ways to identify hazards. A frequently used method is to simply brainstorm what could go wrong. Other tools, like hierarchical holographic modeling (Haimes, et al., 2002) can be used to first create “success scenarios” from which risk scenarios and specific risks can be identified. 

In this formative evaluation, the evaluation team first brainstormed what would be necessary for a successful e-learning Pharmaceutical Cold Chain Management Course (e-PCCMC).  The team then identified actions, events, or situations – the hazards – that could prevent or interfere with e-PCCMC. The list was then condensed based on those hazards that were considered most relevant, and then discussed further using a preliminary risk assessment tool. 

Determining the risks

A preliminary risk assessment (PRA) can be used early on in a project when minimal information is available, or as a screening tool to identify risks that need to be examined more critically using other tools, such as fault tree analysis or failure mode effects analysis (Vesper, 2006). For the purposes of this evaluation, the researcher felt that the PRA would provide an appropriate level of detail. 

For each of the hazards, specific questions were asked to help determine the risk. These included:

1.   What are the potential negative impacts to the learners and the desired course outcomes?  Answers to this question provided examples of the consequences, or harm should the hazard be expressed.

2.   What could cause this unwanted event to occur?  Here, the team identified how the hazard could be expressed.

With this information summarized using a matrix (see Figure 2), the team estimated the likelihood that the hazard would be expressed resulting in the harm, using a scale of low-medium-high (1-2-3) (Column 5). In a similar way, the impact was estimated, again using a scale of low-medium-high (1-2-3) (Column 6). Multiplying these two numbers resulted in a risk score – the higher the number the more risk being present (Column 7). 

The last step of risk assessment is risk evaluation:  deciding on the risks that need to be reduced (Column 8). Generally, these are the high or medium risks that are “treated” through control and mitigation.  Other, low-level risks might be addressed as well if the benefit outweighs the risk-reduction cost. 

Reducing the risks through “treatment”

Risk treatment (ISO, 2009) involves two key concepts:  control and mitigation. Control is aimed at preventing the unwanted event from occurring in the first place; the focus is on reducing the likelihood by targeting the root and contributing causes. Mitigation assumes the unwanted event will occur but aims at protecting the “thing of value” (CSA, 2002). For example, one cannot totally prevent a server crash at a hosting site, but one can take protective measures should that happen. Whenever possible, multiple risk treatment approaches should be taken that have a “layering” of the control and mitigation actions. These are tied to the different causes or mechanisms that were identified. These layers result in a more robust solution should the hazard be expressed.

For each of the risks that were identified, the team identified a risk treatment plan.  In some cases, it was providing information, for example, recommending browsers that were tested (and what browsers are not recommended).  Another example of an identified risk was certain governments not allowing access to a video website.  A mitigation plan was established to pre-make DVDs and send them by DHL courier to course participants when requested by them.  (This actually occurred – actually occurred at the start of the pilot course.  A participant could not access the VIMEO or back-up sites, so the treatment plan – sending a pre-made DVD to him via DHL – was executed.) 

Figure 2. A section of a risk assessment performed using Preliminary Risk Assessment (PRA) worksheet

Monitoring and review

The identified risks were addressed through control and mitigation, however, team members intended to review the assessment at the end of the pilot course to see if the control and mitigation actions were effective and if the likelihood and impact were correctly estimated. Additionally, monitoring was implemented as an ongoing effort to determine if anything that changed that could affect the assessment.  Another aspect of monitoring is to identify any other risks that were not previously identified. 

A formal review of the risk assessment and risk management plan will be performed when the pilot course is completed.  In terms of monitoring, the design team realized two weeks into the course that there was going to be a seasonal time change (from “standard time” to “daylight savings time”) occurring at two different points during the course.  To mitigate the impact, a notice was sent to all participants alerting them to the change.  This event will be included in the listing of risks to be compiled for the next offering of the course.

PRA JIM MENTORS FEB2013 from EPELA on Vimeo.

This is an excerpt from a dissertation chapter from James Vesper. Professors Thomas Reeves and Jan Herrington have contributed to it.