SIMULATING STRESSFUL, EMERGENCY-USE SCENARIOS DURING USABILITY TESTS OF INJECTION DEVICES

EXPERT VIEW

Citation:  Strochlic A, “Simulating Stressful, Emergency-Use Scenarios During Usability Tests of Injection Devices”. ONdrugDelivery Magazine, Issue 97 (May 2019), pp 52-56.

Allison Strochlic highlights the importance of simulating environments that are as realistic as possible when conducting usability tests of emergency-use injection devices, and provides case studies with examples of how to create more realistic tests.

INTRODUCTION

Imagine you are walking in the park on a nice spring day. The sun is shining, the birds are chirping, and kids are running around and playing. Then, next to the playground up ahead, you hear someone yell, “HELP! HELP!” and see a young girl lying on the grass, gasping intensely for air and holding her throat. A few people have gathered around the child, and you run over to join them. A frenzied babysitter begs you and the other bystanders to help, almost sobbing while she stammers: “It’s the first time I’m babysitting for her… I know she’s allergic to bee stings, but I have NO idea how to use this thing… She just got stung, and she’s already having trouble breathing!” She is frantically waving a tube-shaped object in the air: “Can one of you please help me give her this allergy shot? I called 911 but nobody’s here yet, and I’m SO scared!”

Sounds stressful, doesn’t it? This is just one example of the emergency situations that injection devices are used in every day. Human factors (HF) researchers need to find creative ways to simulate stressful scenarios such as the one above when conducting usability tests of emergency-use devices, when basic usability testing approaches might not suffice.

USABILITY TESTING PRIMER

Now, to back up for a moment, usability testing is a common method employed by HF researchers to evaluate the interactive qualities of a given design. The method is most frequently applied to devices with hardware and/or software components, and it involves asking people – specifically, representative end-users – to try using a product or service, observing how it works for them, and seeking feedback regarding various attributes. Such attributes often include usability (whether something is easy or difficult to use), clarity, learnability, and perceived use-safety, depending on the test objectives.

Some consider usability testing to be a “test drive” of sorts, albeit one performed while the “car” is still in development and before it hits dealership sales floors.

There are several types of usability tests, but the most common ones conducted during injection device development are formative and HF validation, the latter of which was previously called summative testing.

A formative test is one conducted iteratively and often as the design is being formed, while an HF validation test is one conducted to validate that the device can be used safely and effectively.

Table 1: Three key usability testing aspects and potential configurations during a formative usability test of an autoinjector.

THE NEED FOR REALISM

Table 1 outlines three key usability testing aspects and how each might be represented in a formative usability test evaluating an autoinjector. The basic approach (described in the right-hand column of Table 1) can be applied to usability tests for a wide variety of injection devices, modified as needed for different drugs and indications, not just emergency use (e.g. including people with diabetes to test an insulin pen-injector).

There are a wide range of devices that have to be used urgently – truly as quickly as possible – to be maximally effective. When it comes to injection devices, those that administer the following medications come to mind:

Figure 1: A typical usability test session of a prefilled syringe, with a human factors researcher and test participant sitting in a room.

• pinephrine (for severe allergic reaction)
• glucagon (for severe hypoglycaemia)
• naloxone (for opioid overdose)
• atropine (for nerve agent exposure).

When conducting usability tests of these and other emergency-use devices, researchers will be well-served by “amping up” the level of environmental and scenario realism, especially when conducting late-stage formative and HF validation tests.

SIMULATION FACTORS

During a typical usability test session, a HF researcher invites test participants into a room, usually one at a time, and engages them in various use scenarios after a brief introduction and informed consent process. It is all very orderly and calm, with the researcher observing a participant’s interactions with a given device and asking questions about the device’s use (Figure 1). However, there are times when more excitement and creative thinking are needed to ensure the usability test – and, the use environment in particular – is as realistic as possible, especially when it comes to testing emergency-use devices.

This need arises not only from researchers’ desire to collect complete, representative data, but also from US FDA and other regulators’ expectations that devices be evaluated and validated within realistic use scenarios and environments.

Figure 2: Sample simulation factors.

Figure 2 lists some of the factors one can consider when it comes to simulating realism and emergency-use scenarios during an injection device usability test.

For example, it’s quite common to have a medley of ambient hospital sounds, such as a beeping monitor, hissing ventilator, conversing visitors, and opening/closing doors, playing in the background as test participants interact with a device intended for clinician use in hospitals. When it comes to lighting, some use scenarios might be presented in normal lighting while others might be presented in dim lighting to represent when a device is used in the middle of the night and/or in an otherwise darkened space. Other factors are simulated less frequently but can still be impactful and relevant. Take air quality, for example. Some injection devices – such as those used to administer atropine and other drugs used to treat a nerve agent attack – are sometimes used in spaces filled with thick fog, which naturally limits someone’s ability to see.

The balance of this article presents two case studies describing how one can implement a subset of the factors listed in Figure 2 during usability tests to simulate stressful, emergency-use scenarios.

CASE STUDY 1: EPINEPHRINE AUTOINJECTOR

As mentioned earlier, a typical injection device usability test session is relatively orderly and calm. A researcher and participant sit at a table and the participant simulates using the device after signing the informed consent form, answering some background questions, and learning more about the test’s focus. When it is time for the “use scenarios,” the researcher might ask the participant to “prepare for and simulate administering an injection with this injection device”.

Contrast that with the following, this time putting yourself in the participant’s shoes: a researcher greets you and asks if you have any final questions about the informed consent form before you sign and walk together to the usability lab’s closed door. Then, the researcher briefly describes the research goals, and asks again if you have any questions (you don’t). With a serious look on her face, and in a hurried tone, she then reads the following: “You are dining in a restaurant. Suddenly, at the next table, a child about 10 years old calls for help. He says his six-year-old brother has a food allergy and is having a bad reaction. The child says that his brother keeps an emergency shot in his backpack. Find the device and give the child’s brother an emergency shot as quickly as possible.”

Figure 3: Participants rush to inject epinephrine into the manikin, which is being used to represent an allergy sufferer.

The researcher opens the door to the usability lab and you see a manikin splayed out on the floor with a backpack next to him. You hear someone wheezing and gasping for air, and there is a young woman standing next to the manikin. She is visibly anxious, and is pleading with you: “Can you help him? Is he going to be OK? Is he going to die?” You start to the dig through the backpack. There’s a water bottle, a notebook, a sweatshirt, a granola bar… and finally, you find plastic tube labelled “AllergyRSQ injector”. You remove the device from the tube, glance at the graphical instructions, and quickly inject the patient’s thigh with the device, injecting the life-saving medication. Figure 3 depicts this test scenario.

Which of the two approaches seems like a more effective way to set the scene for and present an epinephrine autoinjector use scenario – the basic usability test setup or the stressful lab scene? While researchers might implement different simulation factors in a different manner, it would be tough to argue that the second approach does not prevail in terms of realism (see Box 1).

CASE STUDY 2: MILITARY-USE AUTOINJECTOR

For a project focused on a military-use autoinjector, different approaches were used to simulate stressful, emergency-use scenarios during earlier- versus later-stage formative usability tests (see Box 2). During the early stages of development, a formative test focused on ensuring that users could differentiate between, and comfortably handle, two autoinjector prototypes. The objectives later evolved to a rigorous evaluation of the full injection experience, from autoinjector selection and preparation through simulated drug delivery.

As depicted in Figure 4, the level of environmental simulation factors increased in step with the increase in device prototype fidelity and functionality.

For the early-stage formative test, a traditional usability lab was “dressed up” to represent a combat environment. Specifically, a room with neutral décor was transformed by way of a fog machine, flashing lights, looping soundtrack, and a large-scale image of a combat zone projected on one of the large, plain walls. The researcher shared a detailed scenario prompt outside of the lab, as described for the first case study, and standard Mission Oriented Protective Posture (MOPP) gear was provided and donned by the participants at the start of the scenario (see Figure 4, left).

For a later formative test, the level of realism was increased to evaluate device use in a considerably more dynamic (and, arguably, dramatic) use environment. It started with renting a paintball field called Apocalypse City, complete with an ambulance, pick-up truck, police cruiser, and downed plane, as well as several basic structures with “blown-out” windows (see Figure 4, right). For the sounds, overlapping and looping combat sounds playing through high-powered speakers created a realistic cacophony.

Someone with prior military experience served as the victim and, at a predetermined time, she screamed “Help!” and began to cough loudly, running into a small brick structure. The participant quickly grabbed the autoinjectors from the supply kit and headed in after the victim, who was lying on the floor gasping for air in a dim and fog-filled space. The participant quickly administered the injection, often counting the “hold time” aloud and reassuring the victim with a gentle tap on their helmet or shoulder. Carefully positioned cameras, including one worn by the actor, captured the action. (Note: The autoinjector prototypes prepared for testing were needleless, so there was no risk of the actor encountering a needlestick injury.)

BEST PRACTICES

Figure 4: Earlier and later stage usability tests depicting increasing levels of simulation, depicted at left and right, respectively.

In both of the case studies described here, it was evident through observations and participant interviews that the measures implemented to simulate stressful, emergency-use scenarios enabled a more realistic assessment of device use as compared with a typical usability testing approach.

Naturally, there’s not a “one-size-fits-all” approach to selecting and implementing simulation factors. Rather, the best approach for a given usability test will vary depending on the test objectives, as well as a given device’s expected users, use environment, and use scenarios.

Regardless of the type of injection device, the following best practices are recommended:

• Increase simulation fidelity as design matures. It makes sense to implement fewer simulation factors when evaluating initial prototypes in early-stage formative usability tests, and to increase the level of simulation and realism as the design – and usability testing phases – mature, culminating in a relatively high level of realism for HF validation testing.
• Consider all environmental characteristics. Don’t limit yourself to considering only a testing space’s lighting and noise levels, which might be the most obvious factors to modify. Consider all of the factors listed in Figure 2, as well as others that might be warranted based on a given device’s expected use environments.
• Implement several simulation factors. It’s common to implement multiple factors, considering the right mix for a given device. But, don’t go overboard. The goal is to represent the device’s intended use environment – not to win an Oscar for set design.
• Seek feedback on simulation realism. Conducting a pilot test is particularly valuable when it comes to simulating stressful, emergency-use scenarios. Related to the points above and below, you want to hit the “sweet spot” in terms of realism, but not go too far such that you create an unrealistic scenario, or cause participants undue anxiety.
• Seek Institutional Review Board (IRB) review of simulation approach. When conducting usability tests, human factors researchers need to protect human subjects, or test participants. This task becomes even more important when you are simulating stressful use scenarios. On one hand you, are intentionally inducing stress but, on the other hand, you should ensure that no-one becomes upset or overly-emotional due to thetesting scenarios.

CONCLUSION

If a device is tested in an unrepresentative use environment, human factors researchers might miss the opportunity to identify use errors and other interaction problems that, without being identified and mitigated, could hinder immediate, correct use of an injection device needed to administer life-saving medication to someone in dire need. Our goal, therefore, is to test emergency-use devices under conditions that reflect their actual, intended use environments and, as a result, induce the potential stress that would likely accompany device use.