Citation: Newbery M, “How Owen Mumford Carried Out Usability Testing Under Covid-19 Restrictions”, ONdrugDelivery, Issue 120 (May 2021), pp 44–46.
Miranda Newbery describes working with Owen Mumford Pharmaceutical Services on usability testing for a platform autoinjector under covid-19 restrictions.
“Cancelling the usability study was an option but both companies were keen to see if there was a solution that would get them the feedback they really needed.”
When lockdown measures were first introduced in 2020, human factors professionals were faced with the challenge of carrying out usability testing remotely. There is certainly a wide range of human factors tools that can be employed without any contact with end users, such as use risk assessments, expert reviews and device comparisons. Also, known use problems for similar or previous devices can be researched.
However, first-hand user feedback is invaluable and usability testing is a regulatory requirement for most drug delivery devices. It is only when a potential end user tests a prototype that it is possible to truly understand issues with the device, and to make important design decisions. For medical device development to continue, usability studies needed to be reimagined as we entered into the “new normal”. Any new solution still needed to fulfil the required standards for human factors studies, to satisfactorily demonstrate device safety and usability.
“It was important to be able to see elements of the autoinjector up close, and to be able to hear feedback clearly at the start and end of an injection.”
SUCCESS FACTORS
Owen Mumford Pharmaceutical Services was in the latter stages of working on a platform autoinjector when lockdown measures were introduced. Inspired Usability had conducted a large-scale formative study in January 2020 and Owen Mumford wanted to confirm that the subsequent design changes had improved the design and did not cause any unforeseen problems. The best way to do this was via a follow-up formative usability study with potential end users. Cancelling the usability study was an option but both companies were keen to see if there was a solution that would get them the feedback they really needed.
The work began with an assessment of the requirements of an effective usability study to be able to prepare a study design:
“The prototypes were all packed at least three days in advance of the study by engineers wearing PPE – and were kept sealed until needed.”
- Representative participants: to ensure that participants were representative of end users, it was important to brief recruiters thoroughly. As well as the type of device being researched, they had to have a good understanding of what is involved in human factors studies and how they differ from market research. For their part, participants also needed to be briefed fully about what the study would entail so that they could give informed consent.
- Clear protocol and study design: a clear protocol and discussion guide would make it easier to repeat the study multiple times, allowing user actions and behaviours to be compared. The aim was to ensure that the study itself was not too complicated, and to focus on evaluating key parts of the device design.
- A comfortable environment: although the setting had to simulate where the device would be used, participants needed to feel relaxed and not under pressure. Building up a rapport between participants and moderators running the study would put participants at ease so that they felt free to interact with the device as if they were at home (or in their place of work) and to express their views.
- First-hand observation: the moderator, observer and client had to be able to see and hear first-hand what was occurring during the usability sessions. It was important to be able to see elements of the autoinjector up close, and to be able to hear feedback clearly at the start and end of an injection. This also needed to be recorded on video at a high quality to allow for thorough review after the study.
- Device safety: the safety of participants and moderators had to be ensured during the sessions by completing a risk assessment beforehand and putting risk mitigations in place. This was especially important for handling the autoinjector as needles and injectable liquids were involved.
ADAPTING TO COVID-19
“Despite social distancing measures, rapport seemed easy to build as participants appeared to enjoy being out of their homes – and enjoyed the process of testing the device and providing feedback.”
For platform devices, it is important to be as inclusive as possible during testing to cover as many usability problems as possible. In this example, the aim was to test the most challenging cases. The cohort included children, older adults, people with musculoskeletal conditions, people with neurological conditions, people with visual and hearing impairments and healthcare professionals, all with and without injection experience. To increase participation, the chosen venue was in a suburban area with parking, and participants were encouraged to walk or drive rather than use public transport.
In terms of the practical considerations, extra time was needed for each session to allow for cleaning and for the room to be ventilated. There was also additional preparatory work, as the prototypes were all packed at least three days in advance of the study by engineers wearing PPE – and remained sealed until needed. Rather than being handed prototypes for different parts of the study, each participant was allocated a set of trays that contained all the devices they would need (in this case, 15 autoinjectors). The prototypes were numbered and placed in colour-coded sections of the trays so that the moderator could easily direct participants during the session.
Although the general flow of the study remained the same, there were some noticeable changes. The moderator had to describe actions that would normally be done for them, such as handing out prototypes. Both working and non-working prototypes may be used during a study, so it is critical that the correct one is used at each stage. This particular study even included gathering anthropometric data to inform the design input requirements, so the moderator described to participants how to use the equipment. Since needlestick injury was a risk during the study, the moderator was highly vigilant when participants handled the prototypes and there were no incidents.
Despite social distancing measures, rapport seemed easy to build as participants appeared to enjoy being out of their homes – and enjoyed the process of testing the device and providing feedback. To allow for observation of the testing process, a local video company was recruited to help with live video streaming. There were production-equivalent cameras that could transmit high-definition video feeds simultaneously. The picture-in-picture facility showed a close-up view of the autoinjector going into the injection pad, and a wide view of the participants and their interaction with the device. High-quality microphones captured audible feedback and participant responses. Inspired Usability used Microsoft Teams to stream the content so that the Owen Mumford team could ask the moderator questions about participant behaviour during the sessions.
CONCLUSION
Overall, the study was a success. It allowed the Owen Mumford team to move to the design freeze stage, ready for product launch in 2021. The restrictions and changes did not seem to impact the flow of the study, participant recruitment or how the participants interacted with the autoinjectors. The restrictions imposed by covid-19 were a concern at first – but the measures in place soon became second nature, showing people’s ability to quickly adapt to the change in conditions. It is likely that some of these restrictions will stay in place for some time – particularly guidelines on hygiene and infection control, which will become best practice – so the lessons learned over the past year may continue to be useful for future testing.