Formative evaluation of the e-learning course "Pharmaceutical Cold Chain Management"

“Piloting” a course is a well-accepted way to tryout a learning solution with participants in order to make improvements and decide if the learning solution is ready to be launched with a larger, less “controlled” group of learners.  During a pilot, much more data is collected and comprehensively analyzed for formative purposes (i.e., making improvements) than is done during the first full implementation of the course.

The EPELA Pharmaceutical Cold Chain Management e-learning course went through its pilot release starting in March and concluding in early June 2013.  Fifteen people who had previously participated in the actual Pharmaceutical Cold Chain Management on Wheels courses between 2009 and 2012 agreed to participate in the pilot.  Because of work-related and personal reasons, four people withdrew from the course before it ended in June.

Integrated into the course was an online diary where each participant could reflect on what they learned; ideas, concepts and actions that they could take back and apply; and comments (e.g., observations, suggestions) related to that task.  Additionally, a link to an online questionnaire (using the application SurveyMonkey) for each task was provided in order to collect information pertinent to the formative evaluation of the pilot course.  At the end of the course, an hour-long teleconference was arranged with all mentors and participants talking about the course and what it meant to them.

The diaries, questionnaires, emails, and a final group evaluation teleconference were all carefully analyzed.  Participants and mentors made a number of positive comments about what they liked in the course and also what they thought could be improved.  From this data, a number of recommendations were given to the course director/sponsor, Dr. Umit Kartoglu by Jim Vesper, who was both a mentor in the pilot course and the primary evaluation researcher on this project.

In September 2013, several weeks before the start of the first full implementation of the course, Dr. Kartoglu met with Jim Vesper to clarify and implement the recommendations.

Most all of the recommendations could be made by changing instructions, adding more details to activities, and creating newer and better ways for participants and mentors to work together.

Table 1 shows some of the recommendations and actions that were taken to implement them.

Table 1.  Examples of improvements made to e-PCCM course based on recommendations from formative evaluation of pilot course (Mar-Jun 2013)

Recommendations	Actions	Before	After
1.    Review all learning objectives to assure that objectives are aligned to module content and tasks.	All learning objectives were reviewed with many revised to match the instructional intent of the module.  Some of the original objectives were either not covered in the completed modules or did not accurately reflect what was covered in the module.	1.    Identify the major operational components in a pharmaceutical cold chain 2.     Identify deficiencies in given situations using the “good distribution practice” (GDP) guidelines. 3.     For a given a mode of transportation, identify hazards, and assess and identify methods to control the risks to pharma, biopharma, and vaccine products that are consistent with GDP. 4.     For a given a situation, select the appropriate methods and materials for packaging and shipping cold chain products to minimize risk. 5.    For a given a situation, discuss specific requirements to be included in a quality agreement.	1.   Given a situation, propose recommendations to improve compliance with “good distribution practice” (GDP) guidelines. 2.   Given a nonconformance in the transport of pharmaceutical product, analyze data to identify the cause, potential impact to the product, and formulate preventive measures. 3.    Given a list of elements that could be in a quality agreement, justify five elements you consider to be most beneficial. 4.    Given an example of an operational component in a pharmaceutical cold chain, differentiate the practices as to whether or not they reduce risks.
2.     Review all learning objectives in light of activities/tasks that the participants perform and revise as necessary so that the objectives encompass higher levels of cognitive skills.	All learning objectives and activities/tasks were reviewed and verbs were selected to more clearly describe the cognitive skill that the learner was to achieve.	1.    For a given situation, select the appropriate methods and materials needed to monitor temperature and/or humidity for cold chain products and to obtain necessary data for making decisions.	1.   Given a stock situation with different vaccines, various expiry periods and batches and VVM status, decide which products to be dispatched against a requisition order. 2.    Create a decision tree for dispatch of vaccines involving all relevant factors.
3.     Develop a different, more complex activity related to risk assessment and risk management.	A new activity was created for the Hacettepe Hospitals visit.	Initial activity was to review a video and identify the controls in place and the risks that the controls were to reduce.	New activity was to perform a risk assessment on drug storage refrigerators in an isolated area that had only temperature read-outs and local alarms.

Conducting this type of formative evaluation takes time and effort from participants, course mentors, and researchers.  As can be seen from what was done with the EPELA e-PCCM course, formative evaluation can take a learning solution that was very good and make it even better!

e-Pharmaceutical cold chain management beta-course graduates with mentors and designers