Citation: Lawrie-Fussey T, Sheldon L, “We’re Off to See the Wizard: “Use-Like” Prototyping of Digital Concepts”. ONdrugDelivery Magazine, Issue 92 (Dec 2018), pp 12-14.
Tom Lawrie-Fussey and Lucy Sheldon introduce “Wizard of Oz” testing, named after the classic novel and film, whereby experimenters can field test concepts at a very early stage by giving the illusion of a finished product, saving potential costly and time-consuming changes further along the development process.
“In the world of drug delivery, many digital concepts are multi-user, multi-touch-point and must comply with strict regulatory and data protection constraints. This is a far cry from designing, coding and deploying a consumer app…”
It is a simple fact that product development cycles are always being squeezed. In all industries, the cost of an extended time to market means that pressure is applied at every step of the development process. In response, rapid-prototyping techniques have matured quickly and are able to significantly reduce these timescales. Mechanical components, such as the body of an inhaler or the lid of an injection pen, can now be quickly built, improved and refined.
The benefits of these techniques are now widely recognised but the rapid-prototyping of other design functions, such as user interaction, are much less well known. For example, when a novel drug delivery system with new digital interactions is in early-stage development there is a huge benefit in early user testing, as it were, using the digital equivalent of 3D printing.
In theory, digital opportunities do not suffer the same scale of risk and cost as is associated with physical manufacturing. However, in the world of drug delivery, many digital concepts are multi-user, multi-touch-point and must comply with strict regulatory and data protection constraints. This is a far cry from designing, coding and deploying a consumer app. With this complexity comes additional uncertainty and longer development times.
One approach to address this challenge is to use concept testing. Typically, stakeholders are surveyed to provide opinions on the utility and future value of a proposed innovation. As the innovation does not yet exist, their opinions are based on a description of what it would be like once it is made. However, this fails to investigate the user experience or, crucially, how it might influence behaviour. This is vital information in healthcare, compounded because digital services are complex and typically comprise of multiple touch-points and stakeholders.
To use an example from Cambridge Design Partnership’s own work, CDP has been working on a digital education programme aimed at increasing the level of correct inhaler usage in children. CDP wanted to find out whether videos, interactive games or even songs were the most successful way of encouraging best-practice compliance in young inhaler users. The implications go far beyond the patient’s own device. A comprehensive digital service to improve inhaler use must address a range of different drugs and devices, therefore comprising multiple front-end applications. System building blocks include a:
- Secure patient-data hosting solution
- Portal for the payer
- Dashboard for the healthcare professional
- Dashboard for the pharmaceutical company
- Back-end data analytics engine.
In this example, it is hugely advantageous that, in the same way a physical design is frozen before investment in detailed design and manufacturing begins, the digital design is also optimised and pre-validated prior to transitioning into development. If the proposition cannot be tested in a realistic way early on, then late stage changes are inevitable, adding additional, unnecessary cost and time. What is needed is an approach where each stakeholder can assess the part of the system that they will interact with, long before the design is locked down.
WIZARD OF OZ TESTING IN PRINCIPLE
One approach we use on early ideas is to create the illusion in the user’s mind that very early prototypes are in fact final products. This is the theory behind Wizard of Oz testing.
“Have you ever been on a website, seen a fantastic new product and tried to purchase it, only to be added to a waiting list? Well, this is a form of Wizard of Oz testing in action…”
The Wizard of Oz technique enables unimplemented technology to be evaluated by using a human to simulate the response of a system design. The technique is named after L Frank Baum’s novel and the classic film, in which the Wizard is ultimately exposed as a normal man sitting behind a curtain, pulling levers and successfully convincing everyone that he is a powerful magician. The experimenter, like the Wizard, creates a believable illusion of a new device or service. This is then deployed to discover how users interact with an idea and, in so doing, provokes genuine responses.
For example, a user may believe they are speaking to a computer using voice recognition, when, in truth, their words are being typed in manually by an experimenter. The goal here is to observe how a user interacts with a voice recognition system, rather than to measure the effectiveness of the technology that drives it.
This method relies on the generation of what are known as ‘‘use-like’’ prototypes, which do not necessarily need to look like or work like the real thing. The ‘‘use-like’’ prototypes must simply create enough of an illusion to elicit reactions from the people using them that are completely genuine. It is the fidelity of the experience and interaction that is important, not the fidelity of the prototype. For the experiment to work, you simply need to create enough of an illusion to elicit these responses, which then identify the successes and flaws that feed back into the design process.
WIZARD OF OZ TESTING IN PRACTICE
Have you ever been on a website, seen a fantastic new product and tried to purchase it, only to be added to a waiting list? Well, this is a form of Wizard of Oz testing in action. You are faced with the illusion that you can buy an, as yet, non-existent product. In showing an intention to buy, you are added to a list of potential customers, helping the innovators investigate the potential commercial success of their product.
For drug delivery device development, there are many advantages to the Wizard of Oz approach, especially when evaluating early digital concepts. Consider again the system aimed at improving inhaler usage in children. The success of any solution is dependent on how all the stakeholders involved benefit, and when combined, how their behaviour impacts patient outcomes. Utilising “use-like” prototypes and Wizard of Oz testing can shine a light on the aspects of the system that need to be further improved and developed. Figure 1 shows a ‘‘use-like’’ prototype which was created by CDP for a Wizard of Oz research programme to help develop and refine new systems to train patients in correct inhaler technique.
Figure 1: This is a ‘‘use-like’’ prototype developed by Cambridge Design Partnership to trial inhaler use. It is the fidelity of the experience, not the fidelity of the prototype, which is important. The “use-like” prototype consists of a device which collects inhalation profile data and a training app. The app and device were used in combination for inhaler training. The device could be used in isolation to assess the effectiveness of the app and a range of other inhalation training options including current approaches to training as a baseline.
In one example of a Wizard of Oz test protocol, children or adults who are not current inhaler users are trained in how to inhale (without reference to the purpose of the training). This research design assumes that the data collected on the learnability of the inhalation technique is relevant to a first-time inhaler user. Typically, the cohort is split into two groups; one is trained in line with normal practice (to provide a baseline) and the other group trained using the “use-like” prototype and app in Figure 1. However if data on the adherence over time to the new learned behaviour is required, the study becomes more complex and requires patients to be followed and re-tested at realistic intervals in line with known training decay.
Wizard of Oz testing generates quantitative data on the relative potential of the product concept against a baseline, with a general outline of the research design shown in Figure 2. In addition, observation and interviewing uncovers what users love about the idea, as well as highlighting aspects that they may not value, features they struggle with and areas where improvements need to be made.
Figure 2: Example research design.
Given that this process does not involve administering any drugs or therapies, the approach can be undertaken ethically as market research. Appropriate ethics reviews must still be undertaken nonetheless, but these studies can be implemented relatively quickly and economically.
“Wizard of Oz testing of new digital concepts is a powerful tool to optimise and learn about the commercial
potential of digital concepts at a very early stage…”
In parallel with patients themselves, research can be undertaken with stakeholder general practitioners (GPs). They could be presented with various “use-like” representations of information dashboards, in which realistic data visualisation options and anticipated patient scenarios are generated. Faced with believable prototypes GPs can more easily evaluate their desire for new data and researchers can tease out what is most important to them.
Further up the chain again, this research method can address the needs of the pharmaceutical company selling the drug. In the same way as with GPs, but this time a simulated live dashboard showing the market usage and effectiveness of the training programme could be used to inspire innovations that optimise marketing and sales programmes.
Wizard of Oz testing of new digital concepts is a powerful tool to optimise and learn about the commercial potential of digital concepts at a very early stage. The method enables development teams to observe customers interacting with a digital product or service, allowing direct feedback on its potential value in the real world while it is still in the conceptual phase.
It is surely far better to start technical development only when digital concepts have proved themselves in Wizard of Oz-style evaluations. It’s a smart way of ensuring that your new digital device and service will fully deliver its potential.