High-fidelity simulation and virtual reality: an evaluation of medical students’ experiences

With the aims of addressing some of the limitations of high-fidelity simulation and potentially providing a more immersive experience for students, virtual reality technology, in the form of Oxford Medical Simulation (OMS)’s simulation package, 12 has been introduced into the final year medical student simulation sessions at the Hull York Medical School. The simulation package provided by OMS is designed to be used with a head-mounted display, incorporating auditory input, in order to provide a fully immersive experience. The package includes a variety of simulation scenarios, each allowing the student to virtually examine, investigate and to treat a patient in real (simulation) time, with physiological responses to actions and interventions as the scenario progresses. However, understanding of how best to use these evolving virtual reality medical simulation scenarios in undergraduate education is limited. Although simulation using interactive computer programmes presented on a screen has previously been compared with high-fidelity simulation using a manikin, 13 we are not aware of any published studies comparing truly immersive virtual reality using a modern headset to traditional manikin-based simulation. Our aim was to conduct an initial pilot study to compare the two technologies across different domains and consider how the two modalities may be used within the curriculum to optimise their capabilities. We hypothesised that the students would score the manikin based high-fidelity patient simulator higher in areas such as team working skills and communication, with there being little or no difference between the two modalities in other domains, while immersivity may be increased using the virtual reality environment.

While high-fidelity manikins incorporate many features which support the immersivity of the scenario, other aspects such as pallor, heat and the general ‘look’ of the patient are features that are difficult to replicate with a manikin. 4–7 These limitations are well recognised and have contributed towards the creation of a more realistic and immersive experience in medical simulation, in the form of virtual reality. 4 8 Virtual reality is already used in some aspects of healthcare, such as surgical training and as a therapeutic intervention for some forms of mental illness. 9 This evolving form of simulation has the potential to provide a more immersive experience for students and has numerous possible applications in medical education, not least practising emergency patient management. 8 Although there is little research comparing virtual reality to high-fidelity patient simulators in medical education, there is some evidence to suggest that complete immersion, such as that delivered by virtual reality with a head-mounted display, helps to improve knowledge acquisition and conceptual learning. 10 11

Simulation is used in many countries to facilitate teaching and learning in undergraduate and postgraduate medical education. 1 Due to the nearly limitless range of skills potentially recreated in a simulated environment without impacting on patient safety, medical schools across many countries have formally adopted simulation into their medical education curriculum. Virtually all UK medical schools use a variety of simulation techniques to support their curricula, 2 3 with many using at least some of the features of high-fidelity patient manikins such as SimMan 3G.

We used non-parametric statistics for analysis of the quantitative data on the basis that our questionnaire comprised individual Likert items, as opposed to a Likert scale. 17 18 The cross-over pairs were analysed by the Wilcoxon signed-rank test using a nominal level of 5% statistical significance (two-tailed), performed with Stata software. A thematic analysis of the qualitative feedback was performed independently by two investigators. For the purpose of this study, the SimMan 3G simulation was regarded as the ‘gold standard’.

Students completed a questionnaire (available in online supplemental materials ) after each scenario, which aimed to evaluate the two forms of technology at Kirkpatricks’ level one (reaction), 16 by asking students about their experiences of each of the simulation technologies. The questionnaire comprised two sections. The first asked students to rate their perceptions of the simulation technology over nine different domains, including teamworking, immersivity and ease of use, using five-point Likert items. The second part asked open questions to gain qualitative feedback about students’ experiences. In addition, once students had completed the scenario in both technologies, there was a final question asking students to choose which technology they felt was most useful in helping them to prepare for real clinical practice. The questionnaires were pseudonymised using numerical identifiers.

This was an initial scoping study, using a cross-over study design. Twenty students were randomly divided into two groups: Group A and Group B. The study was performed over five small-group simulation sessions. The students in Group A each completed one of the individual simulation scenarios using the OMS virtual reality technology first, while the students in Group B completed an individual scenario using the SimMan 3G first. Each student completed a different scenario to their peers in the same study group. Following a break, the two groups then completed a matched clinical scenario using the other simulation technology ( ).

All students were shown the official introductory video for the OMS virtual reality technology prior to completing the virtual reality scenario. 15 This video introduced the simulated environment, instructed on how to interact within the virtual environment and explored the options available during the scenario. Students were already familiar with SimMan 3G and the simulation suite having used it on a number of previous occasions. One of the key differences identified was the differences in prompts and cues within the scenario. In contrast to prompts being provided by a real nurse assisting in the SimMan 3G scenario, the OMS virtual reality software uses a dropdown selection process for clinical decisions. By using the introductory video prior to virtual reality being introduced, we aimed to address these differences so that participants could solely focus on the response and the underlying clinical reasoning for their decisions.

The research team leading the simulation sessions had all received formal training and had previous experience in delivering teaching sessions using both SimMan 3G and the OMS virtual reality software. Four scenarios were chosen for the study: acute asthma, upper gastrointestinal bleeding, bacterial meningitis and anaphylaxis, based on the pre-designed scenarios in the OMS software, which were predominantly medically based acute care scenarios. The four scenarios reflected core content of the curriculum, and the chosen scenarios were then matched using SimMan 3G.

An a priori sample size calculation was not performed as there was no data on the SD of the outcome measure on which to base a power calculation. A total of 20 participants volunteered for involvement in the research. No students met the exclusion criteria, defined by exclusions outlined in the product licensing of the virtual reality system. 14 Thus all 20 students participated in the research, following written informed consent.

The research took place in the simulation suite at Castle Hill Hospital, Hull University Teaching Hospitals NHS Trust, one of the main clinical teaching bases of the Hull York Medical School. All final year medical students were invited to participate in the research. At the time of the study, students had completed their final year of the MBBS programme. They were in their assistantship period prior to graduation. Recruitment was primarily through online advertisements which were displayed on the students’ virtual learning environment. It included full details of the research, including a participant information leaflet.

Students were asked to comment on the positive aspects of the simulation technology, the negative aspects or limitations of the technology and if they had any further comments. A thematic analysis of the students’ responses showed two main themes: their perceptions of the simulation experience and how the simulation relates to clinical practice ( ). Feedback on the simulation experience was further divided into the common themes of realism, technical aspects, prompting, immersivity of the experience, enjoyment of the scenario and variety of the options available within the scenario. The themes that were identified in relation to clinical practice were communication and handover skills, drug prescribing, ability to perform a systematic assessment, access to guidelines and investigations and gaining practical experience of performing examinations and assessments ( ).

With regards to prompting, 75% of students agreed or strongly agreed that there was a significant amount of prompting during the OMS virtual reality scenario (p<0.01). In contrast, just 35% of students either agreed or strongly agreed that this was the case within the SimMan 3G, with an equal number disagreeing or strongly disagreeing with this statement. Finally, students were asked to decide which simulation technology they felt was most useful overall in helping them to prepare for real clinical practice. Nineteen of the 20 participants answered this question, with a significant majority, 18 out of the 19 students, deciding that the SimMan 3G was superior in achieving this aim.

Facilitating the development of students’ clinical reasoning skills was also explored, with more students agreeing or strongly agreeing that SimMan 3G simulation allowed them to develop these skills (p<0.05). For both categories, the proportions were overall similar between the technologies but with more students strongly agreeing that they could develop clinical reasoning in the SimMan 3G scenario (SimMan 3G 55% vs virtual reality 25%). More students agreed or strongly agreed that SimMan 3G was easy to use when compared with the OMS virtual reality (p<0.05). Students enjoyed using both technologies. All students either agreed or strongly agreed that they enjoyed SimMan 3G scenarios, and 90% of students said they felt this way about their experience of the OMS virtual reality simulation (p>0.05). Although this suggests that student enjoyment of the OMS virtual reality was minimally less, more students (65%) strongly agreed that they enjoyed their experience with OMS virtual reality technology compared with 45% of respondents using the same rating to describe their enjoyment of the SimMan 3G scenarios.

A summary of the Likert item analysis is shown in . Students agreed or strongly agreed that both simulation technologies improved their confidence in performing an ABCDE assessment, but with the SimMan 3G simulation scoring significantly higher (p<0.01). More students agreed or strongly agreed that the SimMan 3G simulation provided a useful learning experience as compared with the OMS virtual reality simulation (p<0.02). A higher proportion of students agreed or strongly agreed that the SimMan 3G simulation provided a realistic experience (p<0.01). Interestingly, 20% of students disagreed that the OMS virtual reality simulation provided a realistic experience. Despite this difference in students’ perceptions towards the realism of the experience, there was no significant difference in reported immersivity (p=0.1). Notably, 15% of students disagreed that during the SimMan 3G simulation, they felt fully immersed in the clinical scenario, rather than being aware of their peers observing. With regards to demonstrating team working skills, the SimMan 3G simulation was regarded more positively than the OMS virtual reality simulation (p<0.05), with 40% disagreeing or strongly disagreeing that the OMS virtual reality simulation permitted the student to demonstrate their team working skills.

Discussion

Previous high-fidelity patient simulator research has shown that this technology can help to improve the acquisition of relevant knowledge and skills and improve student confidence in patient assessments.19–22 In our study, final year medical students experienced both a high-fidelity patient simulator and a new virtual reality simulation environment when training for acute emergency scenarios. The high-fidelity patient simulator was used as the benchmark to assess the utility of the newer and evolving virtual reality technology in day-to-day clinical education. Overall, the students scored SimMan 3G higher across several domains than the newer OMS virtual reality software environment. The students reported that the physical high-fidelity manikin, with haptic feedback, provided a more useful learning experience and improved their confidence more in managing acute emergency scenarios than the virtual reality simulation environment. The students also felt that the high-fidelity manikin was better at providing a familiar and realistic experience, provided less prompting during the scenario, and was also easier to use.

This is understandable for two reasons. First, these final year students had been exposed to SimMan 3G on at least four previous occasions in their final year, whereas it was their first exposure to the virtual reality simulator. Second, their physical presence in a simulated emergency room with a nurse, played by a member of faculty, and a physical manikin is a very familiar environment for medical students. This can be contrasted with the more novel environment of the virtual reality emergency room, avatar patient and avatar nurse. It is therefore interesting to see how highly the virtual reality simulator scored across the domains assessed, even though most of these scores were comparatively lower than the high-fidelity simulator.

The students also perceived SimMan 3G to be better in helping them to develop their team working and clinical reasoning skills, both of which are important for dealing with emergencies in clinical practice.23 This aligns with existing research, which has highlighted the benefits of high-fidelity simulation in developing non-technical skills, such as teamwork.24 Students commented that they found the ‘hands-on’ experience contributed positively to their learning experience, with some students specifically highlighting the benefits of using prescription charts in the scenario for drug prescribing practice. This could only be simulated in the virtual reality environment by using dropdown lists to choose individual drugs or intravenous fluids. These findings are not surprising considering the earlier comments about the two simulation environments. With regards to the negative aspects of SimMan 3G, student feedback highlighted common themes previously recognised in the literature, predominantly a perceived lack of immersivity within the scenario and the inability to recreate aspects that may be important to the clinical picture, such as a rash and skin tone.4 5 7 Ultimately, despite the SimMan 3G high-fidelity patient simulator having many features which are reflective of a real patient, such as palpable pulses and audible breath sounds, the fact that it remains a heavy, relatively static and cold plastic manikin is recognised by students as a significant limitation of this technology.6

The students reported the OMS virtual reality technology to be generally more enjoyable and immersive than the SimMan 3G. The immersive setting that is created within the OMS virtual reality technology system is linked to the fact that, due to the nature of the technology, elements of acute care scenarios that are particularly challenging to recreate outside of the virtual environment may be replicated. For example, the exposure to a myriad of sounds mimicking those of a real-life emergency room.25 Furthermore, and perhaps most importantly, the ‘sick’ avatar patient that is at the centre of the student’s experience of the scenario, and ultimately learning, is more realistic; students are able to interact with their patient, being able to see them talk, look ‘sick’ and also how they change, for the better or for the worse, over time and dependent on their actions. This may be an important factor in relation to suspension of belief, and therefore student engagement and sense of presence in the scenario, which could be more variable and subject to additional external factors within a manikin-based scenario.26 27 Students further commented on the usefulness of having guidelines and results of investigations within the OMS virtual reality system scenarios. The students felt the perceived higher degree of prompting within the virtual reality simulator to be a potentially negative factor. This occurs due to the nature of the virtual reality interface, requiring the use of dropdown menus for selection by the student, whereas in the high-fidelity simulator environment subtle prompting can come verbally from the faculty member present as the nurse. The inability to practise verbal communication and hand-over skills within the virtual environment were other notable areas highlighted by the students. This is likely to be addressed in the future with the evolution of the virtual reality environment when multiple participant scenarios are implemented in multiplayer environments, and emerging research has also suggested that team training using virtual reality was non-inferior to live simulation.28 This application of virtual reality also has the potential for participants not to be physically co-located.

The students felt that SimMan 3G was more helpful in preparing them for real clinical practice, although that they had not yet reached that point. However, the ability to use the OMS virtual reality technology as a learning tool cannot be underestimated. For example, the fact that students felt there was more prompting with the OMS virtual reality system may render this technology more useful to students at an earlier stage of their training. For example, it may provide a means of introducing an immersive experience of managing emergency scenarios to other students, notably those in year 4 of our undergraduate programme. OMS virtual reality technology may provide an excellent opportunity for students to experience, and practice managing, a range of acute emergency scenarios in a more guided and supportive environment. For earlier year clinical students who are developing diagnostic and clinical reasoning skills, but are less confident about patient management, the OMS virtual reality simulation scenarios, with prompting from guidelines and dropdown prescribing options, is likely to be very helpful. In this example, rather than using virtual reality as an educational tool for learning focused on higher cognitive processes such as analysing or evaluating, it could be used for acquisition of knowledge or more less complex cognitive processes, such as remembering and understanding, as described within the revision of Bloom’s taxonomy.29

One important limitation of our study is that our students were already familiar with simulation using the SimMan 3G and this may have impacted on the students’ perceptions of both technologies. For example, familiarity with SimMan 3G could easily have influenced question responses, for example, about ease of use or usefulness, when compared with the less familiar virtual reality environment. Most of the students were naïve to the OMS virtual reality technology and despite orientation using the introductory video from OMS about the virtual reality environment, there may have been an impact on responses related to differential familiarity. We directly supervised the virtual reality simulation in an identical way to our procedures for performing SimMan 3G simulation, including debriefing. We did not use the inbuilt debriefing tool within the OMS technology, which can be used by students without direct supervision to practice the scenarios and increase their scores, as assessed by the internal algorithm of the virtual reality software. Enhanced familiarity with the virtual reality environment may well increase the experiential benefit of this form of technology and benefit from the computer derived performance metrics permitting students to track their progress. An additional limitation of our study was that the outcome data reflected only student perceptions of the simulation technologies, reflecting Kirkpatricks’ level one, as opposed to data that demonstrate achievement of learning outcomes or Kirkpatricks’ level two data.16 While student perceptions of learning are an important component of the educational experience, it is recognised that a more robust evaluation of how the technologies help to achieve or assess specific competencies would provide more useful information for educators to help inform decisions about how best to use different forms of simulation.

As an initial exploratory study, we recognise that the results of this study are not definitive, and that further work may be required in this area across a number of disciplines in order to provide more robust data. Further evaluation of the simulation technologies and the experiences of final year medical students that are non-naïve to both technologies is currently in progress by our research team. The next steps of our research aim to not only address this limitation of this study, but also to further explore how the two technologies can address different aspects of the medical curriculum, such as prescribing and interpreting clinical investigations.

From a practical perspective, the OMS virtual reality kit can also be easily transported between clinical teaching sites. This is of particular value given that teaching at the Hull York Medical School and most other UK medical schools, occurs over a range of hospital sites. Similarly, the OMS technology software has been designed to allow students to log into the scenarios from their personal computers. Although this facility is not currently being used by our students, it is important to recognise that this technology can potentially be utilised in this way. With an increasing focus on self-directed and technology-enhanced learning, the value of providing students with an opportunity to practise the OMS technology emergency scenarios many times, and at their leisure, cannot be underestimated.