You are here
Pilot Validation Study of the European Association of Urology Robotic Training Curriculum
European Urology, Volume 68, Issue 2, August 2015, Pages 292 - 299
The development of structured and validated training curricula is one of the current priorities in robot-assisted urological surgery.
To establish the feasibility, acceptability, face validity, and educational impact of a structured training curriculum for robot-assisted radical prostatectomy (RARP), and to assess improvements in performance and ability to perform RARP after completion of the curriculum.
Design, setting, and participants
A 12-wk training curriculum was developed based on an expert panel discussion and used to train ten fellows from major European teaching institutions. The curriculum included: (1) e-learning, (2) 1 wk of structured simulation-based training (virtual reality synthetic, animal, and cadaveric platforms), and (3) supervised modular training for RARP.
Outcome measurements and statistical analysis
The feasibility, acceptability, face validity, and educational impact were assessed using quantitative surveys. Improvement in the technical skills of participants over the training period was evaluated using the inbuilt validated assessment metrics on the da Vinci surgical simulator (dVSS). A final RARP performed by fellows on completion of their training was assessed using the Global Evaluative Assessment of Robotic Skills (GEARS) score and generic and procedure-specific scoring criteria.
Results and limitations
The median baseline experience of participants as console surgeon was 4 mo (interquartile range [IQR] 0–6.5 mo). All participants completed the curriculum and were involved in a median of 18 RARPs (IQR 14–36) during modular training. The overall score for dVSS tasks significantly increased over the training period (p < 0.001-0.005). At the end of the curriculum, eight fellows (80%) were deemed able by their mentors to perform a RARP independently, safely, and effectively. At assessment of the final RARP, the participants achieved an average score ≥4 (scale 1–5) for all domains using the GEARS scale and an average score >10 (scale 4–16) for all procedural steps using a generic dedicated scoring tool. In performance comparison using this scoring tool, the experts significantly outperformed the fellows (mean score for all steps 13.6 vs 11).
The European robot-assisted urologic training curriculum is acceptable, valid, and effective for training in RARP.
This study shows that a 12-wk structured training program including simulation-based training and mentored training in the operating room allows surgeons with limited robotic experience to increase their robotic skills and their ability to perform the surgical steps of robot-assisted radical prostatectomy.
Keywords: Curriculum, Radical prostatectomy, Robotics, Simulation, Training.
The concept of surgical training has been evolving in the last decade from the traditional concept of “see one, do one, teach one” towards better defined and standardized methodologies for surgical education based on the development of skill-based curricula , , , , and . Furthermore, the development and diffusion of surgical robotic platforms are increasingly supporting the development, use, and validation of simulation-based training methods ranging from bench-top synthetic models, animal, and cadavers to high-fidelity virtual training platforms , , and . Simulation-based training should be an essential part of surgical training programs to significantly improve the technical and nontechnical skills of trainees, shorten their learning curves for different procedures, and improve surgical safety  and .
Nevertheless, training for robotic techniques remains mainly unstructured. There has been a recent call by various training bodies for the development of well-organized educational curricula to increase preclinical exposure and of validated assessment tools that allow constructive feedback for performance improvement. These curricula, as well as proficiency-based credential processes, are important for improving patient safety and surgical outcomes in urological surgery  and .
On the basis of these considerations, the European Association of Urology (EAU) Robotic Urologic Section (ERUS) has designed and developed a structured training program and curriculum in urology that focuses on robot-assisted radical prostatectomy (RARP). The aim of the present study was to assess the feasibility, acceptability, face validity, and educational impact of this curriculum, and to assess improvements in performance and ability to perform RARP after completion of the curriculum.
2. Materials and methods
2.1. Study design and participants
This was a longitudinal prospective study using quantitative observational measures. The participants were ten international fellows training in robotic surgery provided by major teaching European institutions under the recommendation of an expert mentor.
The curriculum was developed based on an expert panel discussion  and was used for training of fellows. The key components of the curriculum include: (1) e-learning, (2) an intensive week of structured, simulation-based training (virtual reality synthetic, animal, and cadaveric platforms), and (3) supervised modular training in RARP ( Fig. 1 ).
The overall study duration was 12 wk. After evaluation of baseline experience, the fellows underwent e-learning using the e-module developed by the ERUS expert panel  and observed and assisted in live surgery for 3 wk. The participants then underwent an intensive week of structured, simulation-based laboratory training including virtual reality simulation (da Vinci surgical simulator, dVSS) and dry and wet laboratory simulation platforms (synthetic, animal, and cadavers) (Supplementary Table 1). Following this, the fellows participated into a modular training program under supervision, which involved progressive, proficiency-based training through surgical steps with increasing levels of complexity (Supplementary Table 2)  . The fellows continued the training until they carried out a full RARP procedure, which was assessed by their mentors and video-recorded for review and evaluation of performance by blind assessors.
2.4. Study outcomes
The outcomes of interest were (1) the feasibility, acceptability, face validity, and educational impact of the curriculum  and (2) improvements in performance and ability to perform RARP following completion of the curriculum. Face validity is the extent to which the learning and assessment environment resembles the situation in the real world  .
2.5. Evaluation of outcome parameters
Feasibility, acceptability, face validity, and educational impact were assessed using quantitative surveys that were developed and validated according to the expert opinions of robotic surgeons.
The technical skills of the participants were assessed via inbuilt validated assessment metrics on the dVSS. The specific skills included moving the camera and clutching, manipulating the endowrist, use of energy and dissection, and needle driving. The score at baseline and on final assessment were compared to determine the improvement in basic robotic skills.
Following successful completion of the modular training, the mentors evaluated the procedural skills of the fellows in performing RARP using the validated Global Evaluative Assessment of Robotic Skills (GEARS) score (Supplementary Table 3)  . The mentors also assessed the quality of the surgical skills for each surgical step using a RARP procedure-specific scoring scale (Supplementary Table 4) ranging from 1 to 5, for which ≥3 was considered safe.
Videos of the final RARP procedures performed by each fellow were further assessed by blinded, expert robotic surgeons using a generic dedicated scoring criterion for each procedural step (Supplementary Table 5). This score ranged from 4 to 16, and ≥10 was considered safe. The videos of each surgical step were assessed by the same two independent reviewers for all participants. The scores obtained by the fellows were compared with those assigned by the same blinded reviewers to the performance of two expert robotic surgeons to establish the construct validity of the assessment. Construct validity is the extent to which a test is able to discriminate between various levels of expertise  .
2.6. Statistical analysis
Descriptive statistics were performed for the available variables. Categorical variables are reported as frequency and percentage, and continuous variables as median and interquartile range (IQR) or mean and standard deviation, as appropriate. Mean values among groups were compared using the Student t test and analysis of variance, as appropriate. Statistical significance was set at p ≤ 0.05. The statistical analysis was carried out using SPSS version 20 (IBM Corp., Armonk, NY, USA).
The characteristics and previous robotic experience of the participants are reported in Table 1 . Most participants had minimal or no previous experience of simulation-based training. The median times of involvement as a table assistant and a console surgeon at baseline were 9.5 mo (IQR 5.75–16 mo) and 4 mo (IQR 0–6.5 mo), respectively.
|Median age, yr (IQR)||35 (31–36)|
|Training stage, n (%)|
|Involvement in robotic surgery as table assistant|
|Median time, mo (IQR)||9.5 (5.75–16)|
|Median cases, n (IQR)||50 (29.5–175)|
|Involvement in robotic surgery as console surgeon|
|Median time, mo (IQR)||4 (0–6.5)|
|Median cases, n (IQR)||12 (0–24)|
|Median experience of virtual reality simulation, h (IQR)||0.5 (0–6.5)|
|Median experience of dry laboratory, h (IQR)||0 (0–8)|
|Median experience of wet laboratory, h (IQR)||0 –|
IQR = interquartile range.
All participants completed the required e-learning module and passed the final test for assessment of theoretical knowledge. All fellows observed and assisted in the recommended number of procedures (>12 cases) during the first 3 wk of the curriculum, participated successfully in all activities during the intensive week of laboratory training, and completed the 8 wk of supervised modular training. The median number of RARPs they were involved in as console surgeons during modular training was 18 (IQR 14–36).
The overall scores obtained by participants for performance of four different dVSS tasks at baseline, during the training program (weeks 4 and 5), and at the end of the curriculum are reported in Figure 2 . For all exercises the overall score significantly increased over the training period (p < 0.001–0.005).
The GEARS scale was used by mentors to evaluate the final RARP performance of fellows ( Fig. 3 A). Good to excellent scores were obtained by 80–100% of trainees for their depth perception (median 5, IQR 4–5), bimanual dexterity (median 4.5, IQR 4–5), efficiency (median 4, IQR 3.75–5), force sensitivity (median 4, IQR 4–5), autonomy (median 4, IQR 3.75–5), and robotic control (median 5, IQR 3.75–5) skills. A procedure-specific scale was used to score the performance of fellows for each RARP surgical step ( Fig. 3 B).
Eight trainees (80%) were considered by their mentors able to perform a RARP independently, safely, and efficiently on completion of the curriculum. Three (30%) were considered able to perform a complex RARP independently, safely, and effectively.
The blinded video-based assessment results for RARP steps performed by each fellow and two robotic experts are shown in Table 2 . The fellows achieved an average score that was considered safe (≥10) for all surgical steps. The highest average scores were obtained for bladder detachment and urethrovesical anastomosis. When the scores for all procedural steps were assessed, eight fellows (80%) reached an average score ≥10. The two participants who did not reach an average sufficient score were residents. The robotic experts significantly outperformed the fellows overall (mean score 13.6 vs 11) and for all RARP steps except bladder detachment and endopelvic fascia incision, confirming the construct validity of the assessment ( Fig. 4 ).
|Bladder detachment||Endopelvic fascia incision||Ligation of dorsal vein complex||Bladder neck incision||Dissection of vasa and seminal vesicles||Preparation and section of prostatic pedicles||Dissection of neurovascular bundles||Apical dissection||Urethrovesical anastomosis||Mean|
NP = not performed.
The results for quantitative surveys completed at the end of curriculum are reported in Figure 5 . All fellows gave an excellent overall evaluation of the training program and felt that the curriculum was very effective in improving their console exposure, their basic robotic skills, and their ability to perform RARP.
This is the first study that incorporates and validates different components of a training curriculum for robot-assisted surgery at a multi-institutional level. The study demonstrates that a 12-wk structured training program including theoretical e-learning, laboratory training, and modular training in the operating room is feasible, acceptable, and effective in improving the technical robotic skills and ability of young surgeons with limited previous robotic experience to perform the surgical steps of RARP. The study also demonstrates the face validity of this curriculum.
In the last few years there has been growing interest in the field of surgical education, especially in minimally invasive laparoscopic and robotic surgery , , , and . However, curricula for training in robotic urologic procedures have not yet been standardized and the optimal integration of simulation-based training in surgical training programs is not clearly defined or evidence-based. The definition of curricula for each surgical procedure and their validation and progressive implementation are important for accreditation of surgeons and teams for each specific intervention, with the ultimate aim of improving surgical safety and patient outcomes  and .
To date, three basic curricula for training and assessment of robotic surgeons have been developed and reported in the literature , , and . It has been shown that they are valid, feasible, and effective in significantly improving basic robotic surgery skills. However, these curricula do not include modular training in the operating room and they have not been validated for specific procedures.
On the basis of these considerations, the ERUS board members designed and proposed a curriculum for RARP including simulation-based and modular console training with an expert mentor. The curriculum content was finalized in accordance with the expert opinion of robotic surgeons across various regions to ensure content validation  . The program is a step forward from existing approaches because it includes all the simulation training modalities available (virtual reality simulation, bench-top models, live animal surgery, and cadaveric procedures) combined with clinical training in the form of a mini-fellowship. Fellowships are actually considered a key component of training for complex urologic procedures such as RARP  .
An ideal training program should be feasible, acceptable, valid, and economically sustainable, with an effective educational impact  . Our results show that the EAU robotic training curriculum is feasible and acceptable, as all participants were very satisfied with the program and would highly recommend it to other colleagues. All modules of the curriculum were found to be useful, although the fellows subjectively found the virtual reality simulation training, the dry and wet laboratories, and the cadaver training particularly valuable. These data suggest that all these elements should ideally be integrated in training programs, although there are cost, ethical, and regulatory issues to take into account. The centralization of simulation training in an intensive week at a single, fully equipped training centre was well perceived and proved to be effective, as previously demonstrated in other studies  .
An effective simulation-based training program represents the ideal background for clinical training in the operating room. The concept of modular training proposed by Stolzenburg et al  for laparoscopic radical prostatectomy was adopted in the curriculum for progressive, proficiency-based training of a fellow through steps of increasing levels of difficulty in a surgical procedure. This strategy has the potential to overcome the problems of teaching complex surgical procedures, allowing safe training of surgeons with limited expertise and potentially accelerating their learning curve. The importance of proper mentoring for effective modular training has recently been highlighted  .
This pilot study showed that the EAU robotic training curriculum can effectively improve the basic robotic skills of participants, as demonstrated by the significant improvement in overall scores achieved by fellows at the end of the curriculum for all dVSS tasks. The first weeks of simulation training were found to be particularly effective in this respect, potentially allowing participants to optimize the results of subsequent training in the operating room. Importantly, at the end of the curriculum the majority of fellows were able to perform RARP with good or acceptable technical quality, as reflected by scores from mentors and blinded reviews of video-recorded surgeries. The two participants who were not deemed able to perform RARP safely and efficiently by the end of the training program were residents. This may indicate that the curriculum is more likely to be effective for urologists who have already completed their basic training in residency programs.
The curriculum also provided constructive feedback on the performance of fellows for individual procedural steps. Overall, the fellows were able to reach similar performance levels compared to expert surgeons for the easiest RARP steps, such as bladder detachment and endopelvic fascia incision. However, the experts achieved significantly better scores for challenging parts of the procedure, confirming the construct validity of the assessment. Fellows who did not reach sufficient scores for specific surgical steps will need further training before being deemed able to safely perform RARP independently.
Finally, the results indicate that the curriculum has a good educational impact, as well as face validity, defined as the extent to which the program is subjectively viewed as delivering the desired goal. In fact, the majority of the participants felt that the curriculum significantly improved their robotic skills and their ability to perform different RARP steps. However, further comparative studies and, ideally, randomized controlled trials will be needed to assess whether this curriculum is superior to traditional nonstructured training for RARP.
This study has some limitations. First, the number of participants was limited. Second, although the participants had limited previous robotic experience, they were not all completely novice to console surgery, so the results for the training program may be overestimated. The experience level of mentors was not factored in the analysis, but they were all high-volume robotic surgeons at teaching institutions. Third, modular training was not standardized and the console exposure of the participants was variable. Fourth, assessment of the technical skills of the fellows using GEARS and the procedure-specific scoring scale at the end of the training program was performed by their mentors. However, to standardize the assessment process and minimize bias due to nonblinded assessment, the mentors were informed about and educated in the use of the scoring systems. Fifth, the performance of fellows in RARP video clips was assessed using a new scoring tool. The contents of this scoring criterion were developed and validated on the basis of expert opinion. Sixth, a 3-mo training program may be too short for some fellows to achieve sufficient skills to adequately perform complex parts of the procedure. Finally, there was no specific training or proper pre- and post-training assessment of nontechnical skills.
This study represents the first step towards the definition of an ideal training program for RARP and of criteria for accreditation of surgeons for this procedure. Further studies need to be carried out to address the issues associated with certification and recertification using training curricula.
This study establishes the effectiveness of the first structured training curriculum for robot-assisted surgery that integrates simulation-based training in dry and wet laboratories, and modular training in the operating room with expert mentorship. The study shows that the 12-wk curriculum is valid, feasible, and acceptable, and has a good educational impact, allowing participants to improve their basic robotic skills and their ability to perform the surgical steps of RARP. Further studies are needed to better define the ideal length and structure of these training programs.
Author contributions: Alexandre Mottrie had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.
Study concept and design: Mottrie, Dasgupta, van der Poel, Ficarra, Volpe.
Acquisition of data: Volpe.
Analysis and interpretation of data: Volpe, Ahmed.
Drafting of the manuscript: Volpe, Ahmed.
Critical revision of the manuscript for important intellectual content: Mottrie, Dasgupta, van der Poel, Novara, Ficarra.
Statistical analysis: Volpe.
Obtaining funding: None.
Administrative, technical, or material support: None.
Other (specify): None.
Financial disclosures: Alexandre Mottrie certifies that all conflicts of interest, including specific financial interests and relationships and affiliations relevant to the subject matter or materials discussed in the manuscript (eg, employment/affiliation, grants or funding, consultancies, honoraria, stock ownership or options, expert testimony, royalties, or patents filed, received, or pending), are the following: None.
Funding/Support and role of the sponsor: None.
Acknowledgments: We would like to thank the ten participants (F. Audenet, A. Briganti, M. Brown, V. De Marco, M. Gan, M. Janssen, M. Oderda, R. Navarro, R. Sanchez Salas, and E. Wit) and their respective mentors and teaching institutions (R. Sanchez Salas, Institut Montsouris, Paris, France; F. Montorsi, San Raffaele Hospital, Vita Salute University, Milan, Italy; P. Dasgupta, Guy's Hospital, Kings College, London, UK; W. Artibani, University of Verona, Verona, Italy; G. De Naeyer, OLV Hospital, Aalst, Belgium; M. Stöckle, University of Saarland, Homburg/Saar, Germany; T. Piechaud, Clinique Saint Augustin, Bordeaux, France; A. Ruffion, Centre Hospitalier Lyon Sud, Hospices Civils de Lyon, Lyon, France; P. Wiklund, Karolinska University Hospital, Stockholm, Sweden; and H. van der Poel, The Netherlands Cancer Institute, Amsterdam, The Netherlands) for their contribution, enthusiasm, and dedication to this project. We would like also to acknowledge the contributions of the ERUS board members ( http://www.uroweb.org/sections/robotic-urology-erus/?no_cache=1 ) and the reviewers of the surgical videos (C. Assenmacher, N. Buffi, G. D’Elia, P. Dekuiper, N. Doumerc, V. Ficarra, S. Klaver, D. Murphy, K. Palmer, C.H. Rochat, C. Vaessen, B. van Cleynenbruegel, C. Wagner, J. Walz, C. Wijburg, and J. Witt).
-  R.L. Kneebone. Practice, rehearsal, and performance: an approach for simulation-based surgical and procedure training. JAMA. 2009;302:1336-1338 Crossref
-  D.J. Scott. Patient safety, competency, and the future of surgical simulation. Simul Healthc. 2006;1:164-170 Crossref
-  A.P. Stegemann, K. Ahmed, J.R. Syed, et al. Fundamental skills of robotic surgery: a multi-institutional randomized controlled trial for validation of a simulation-based curriculum. Urology. 2013;81:767-774 Crossref
-  R. Smith, V. Patel, R. Satava. Fundamentals of robotic surgery: a course of basic robotic surgery skills based upon a 14-society consensus template of outcomes measures and curriculum development. Int J Med Robot. 2014;10:379-384 Crossref
-  J.Y. Lee, P. Mucksavage, C.P. Sundaram, E.M. McDougall. Best practices for robotic surgery training and credentialing. J Urol. 2011;185:1191-1197 Crossref
-  L.M. Sutherland, P.F. Middleton, A. Anthony, et al. Surgical simulation: a systematic review. Ann Surg. 2006;243:291-300 Crossref
-  N.E. Seymour, A.G. Gallagher, S.A. Roman, et al. Virtual reality training improves operating room performance: results of a randomized, double-blinded study. Ann Surg. 2002;236:458-463
-  K. Ahmed, M. Jawad, M. Abboudi, et al. Effectiveness of procedural simulation in urology: a systematic review. J Urol. 2011;186:26-34 Crossref
-  S.B. Issenberg, W.C. McGaghie, E.R. Petrusa, D. Lee Gordon, R.J. Scalese. Features and uses of high-fidelity medical simulations that lead to effective learning: a BEME systematic review. Med Teach. 2005;27:10-28
-  J. Brewin, K. Ahmed, B. Challacombe. An update and review of simulation in urological training. Int J Surg. 2014;12:103-108 Crossref
-  K.C. Zorn, G. Gautam, A.L. Shalhav, et al. Training, credentialing, proctoring and medicolegal risks of robotic urological surgery: recommendations of the Society of Urologic Robotic Surgeons. J Urol. 2009;182:1126-1132 Crossref
-  Khan RS, Ahmed K, Mottrie A, et al. Towards a standardised training curriculum for robotic surgery: a consensus of an international multidisciplinary group of experts. Proceedings EAU Robotic Urology Section Congress. Stockholm 2013.
-  http://www.uroweb.org/index.php?id=466&cid=3&auth=1
-  J.U. Stolzenburg, H. Schwaibold, S.M. Bhanot, et al. Modular surgical training for endoscopic extraperitoneal radical prostatectomy. BJU Int. 2005;96:1022-1027 Crossref
-  K. Ahmed, M. Jawad, P. Dasgupta, A. Darzi, T. Athanasiou, M.S. Khan. Assessment and maintenance of competence in urology. Nat Rev Urol. 2010;7:403-413 Crossref
-  A.C. Goh, D.W. Goldfarb, J.C. Sander, B.J. Miles, B.J. Dunkin. Global evaluative assessment of robotic skills: validation of a clinical assessment tool to measure robotic surgical skills. J Urol. 2012;187:247-252
-  K. Foell, A. Finelli, K. Yasufuku, et al. Robotic surgery basic skills training: evaluation of a pilot multidisciplinary simulation-based curriculum. Can Urol Assoc J. 2013;7:430-434
-  D. Hay, M.S. Khan, H. Van Poppel, et al. Current status and effectiveness of mentorship programmes in urology—a systematic review. BJU Int.. 2014;
-  M. Shamim Khan, K. Ahmed, A. Gavazzi, et al. Development and implementation of centralized simulation training: evaluation of feasibility, acceptability and construct validity. BJU Int. 2013;111:518-523 Crossref
a OLV Vattikuti Robotic Surgery Institute, Aalst, Belgium
b Division of Urology, University of Eastern Piedmont, Novara, Italy
c MRC Centre for Transplantation, Kings College London, Guy's Hospital, London, UK
d Department of Experimental and Clinical Medical Sciences, Division of Urology, University of Udine, Udine, Italy
e Department of Surgical, Oncological and Gastrointestinal Sciences, Division of Urology, University of Padua, Padua, Italy
f Division of Urology, The Netherlands Cancer Institute, Amsterdam, The Netherlands
g Division of Urology, Onze-Lieve-Vrouw Hospital, Aalst, Belgium
⁎ Corresponding author. OLV Vattikuti Robotic Surgery Institute, Department of Urology, Onze-Lieve-Vrouw Hospital, Moorselbaan 164, 9300 Aalst, Belgium. Tel. +32 53 724378; Fax: +32 53 724411.
© 2014 European Association of Urology, Published by Elsevier B.V.