Cumulative sum analysis of the robotic learning curve in the surgical management of malignant pelvic neoplasms

Introduction

For most types of pelvic cancer, the standard staging and treatment surgeries consist of specific organ resection associated with regional lymph node dissection (1-4). Such surgeries are traditionally approached by laparotomy through a midline or transverse incision. Minimally invasive surgery through laparoscopy or robotic system is an alternative approach associated with fewer complications, shorter hospitalization and faster recovery with similar oncological results for rectal, gynecological and prostate cancer (5-10).

Robotic-assisted surgery (RAS) has the potential to overcome the obstacles of standard laparoscopy by introducing wristed instruments which allow the surgeon to regain the two lost degrees of freedom. The value of using six degrees of freedom is of particular relevance when operating in a narrow space such as the pelvis (11,12). These advancements might facilitate surgeons to progress quickly along the learning curve (LC). The learning process of a new surgical skill can be defined as the time and number of procedures required for an individual surgeon to achieve proficiency in a specific procedure (13).

One method to evaluate the surgical skills acquisition is based on sequential monitoring of a cumulative performance over time. The cumulative sum (CUSUM) method, first described by Page in 1954 (14), was originally devised for monitoring performance and detecting areas for improvement in the industrial sector. With several developments and adaptations, it has emerged as a suitable method for monitoring healthcare outcomes (15-19) and was adopted by the medical profession in the 1970s to analyze the LC of surgical procedures (20).

The objective of this study is to evaluate and compare the LC of different oncological pelvic specialties in RAS using the CUSUM methodology. Body mass index (BMI), conversion rate (CR) to open surgery and estimated blood loss (EBL) were also studied in order to determine if there is a correlation with the LC. We believe that these data can guide the development of a more individualized proctoring and learning program for each specialty.

Methods

Patients and study design

Between January 2012 and June 2016, 395 consecutive patients were operated on by using the DaVinci Si HD Robotic System (Intuitive Surgical Inc., Sunnyvale, CA, USA) by the Urology, Gynecology and Abdominopelvic Departments of a tertiary referral cancer center in Brazil. This period comprehends the implementation and learning phase of robotic surgery in our institution. All patients who underwent robot-assisted laparoscopic total mesorectal excision (RALTME) for rectal cancer, robotic-assisted laparoscopic hysterectomy (RALH) with pelvic lymphadenectomy for endometrial cancer, robotic-assisted laparoscopic radical hysterectomy (RALRH) with pelvic lymphadenectomy for cervical cancer and robotic-assisted laparoscopic radical prostatectomy (RALRP) for prostate cancer were included in the analysis. All other types of surgeries and disease were excluded. We included only cases operated by robotic surgeons who had at least 20 robotic surgeries during the analysis period. All surgeons had great experience in open and laparoscopic procedures. They were trained by observations of RAS, simulator training for at least 20 hours and an experienced robotic surgeon proctored the first ten robotic cases for each surgeon. These proctored surgeries were not included in the analysis. Cases operated by surgeons who had fewer than 20 robotic surgeries were excluded. These criteria were used since LC analysis of few cases could not be performed accurately. Surgeons were described by number in order to avoid identifications.

Data were extracted from our prospectively maintained RAS database, which contains information regarding patient demographics, diagnosis, clinical stage, and preoperative assessment. The surgical steps analyzed and considered relevant to reflect the surgical LC were: docking time (DT) which was defined as the time required to move the robot and securing the robotic arms to the corresponding port sites; surgeon console time (SCT) which was defined as the actual time the surgeon spent at the robotic console during the procedure, which directly corresponded to the robotic portion of the procedure; and total operation time (TOT) which was defined as the time between the first skin incision and the last port closure. All times were precisely clocked by the robotic assistant nurse. Since each surgeon’s experience began at a different time during the study period and had a different interval between cases, time was defined as the number from the first to the last case that each surgeon performed in the cohort. Pre-operative and intraoperative parameters including DT, SCT, TOT, BMI, EBL, and CR were analyzed.

The study was conducted in accordance with the regulations of the local ethics committee and was approved by the institutional review board. All involved surgeons in this study signed the research informed consent. Patient informed consent was not necessary since it was a retrospective analysis, no intervention was performed and the subject of the study was the surgeon’s operative times.

LC analysis

To access the LC, operative times of every single procedure were analyzed with respect to the chronological order. Cases were grouped by type of surgeries and by surgeons. All specific surgical step times were analyzed by linear regression and cumulative sum (CUSUM) methods.

Linear regression was performed with a simple linear regression model (Y = β0 + β1X). This method was used to estimate the relationship between the number of procedures and improvement of surgeon operative times and to verify correlations between BMI, CR, EBL and the LC.

The CUSUM method was used for quantitative assessment of the LC. Basically, CUSUM is the running total of differences between the individual data points and the mean of all data points (21). Cumulative sum analysis transforms raw data into the running total of data deviations from their group mean, enabling investigators to visualize the data for trends not discernable with other approaches. It recognizes the importance of time and experience in clinical practice and allows the identification of improvements by standard statistical methods (15-23).

Finally, competency of the procedure was defined as the first turning point of the curve plateau and proficiency was defined as the turning point at which the slope of the curve becomes less steep.

Statistical analysis

All statistical analyses were performed using IBM SPSS Statistics for Windows version 19 software (SPSS, an IBM company, Chicago, IL, USA) and R version 3.0.1 (R Foundation for Statistical Computing). Kolmogorov-Smirnov test for adherence to the normal distribution curve was used to assess distribution curve symmetry, identifying normal distribution only for the variable age. The quantitative variables were expressed in median, with a lowest and highest value, except for age, which was expressed in mean standard deviation. The categorical variables were expressed in percentages. Comparison of the medians between two groups was done by nonparametric Mann-Whitney test and to compare median between three or more groups we used Kruskal-Wallis nonparametric test. The associations between categorical variables were done by Fischer’s exact test. Comparison between surgical times and BMI were made by distributing all patients into two groups, Group 1: BMI <25 kg/m² and Group 2: BMI ≥25 kg/m². The BMI cutoff was set at 25 kg/m² based on the cutoff for normal and overweight according to the WHO Classification (24). Correlations between BMI and operative times were accessed by the nonparametric Spearman’s rank correlation coefficient. P values were derived from two-tailed tests and data differences between groups were considered statistically significant at the level of P<0.05.

Results

During the study period, 395 RAS was performed by 10 surgeons of the Urology, Gynecology and Abdominopelvic Departments. Out of these, 343 RAS and 7 surgeons were selected to be analyzed. Patient demographics, surgical procedures, enrolled surgeons, operative data and times are summarized in Table 1. Comparison of patient characteristics and operative times showed that all parameters differ significantly between groups (P<0.000), however all patients characteristics were homogenous inside each surgical group.

One hundred and three RALTME were performed by 3 rectal surgeons, 41 procedures by Surgeon 1, 25 procedures by Surgeon 2 and 37 procedures by Surgeon 3. There were 4 (3.9%) conversions to open surgery in this group. Surgeon1 had 3 (7.3%) conversions to laparotomy, 1 related to grade III obesity (BMI 44) and 2 related to technical difficulty. Surgeon3 had one conversion related to technical difficulty. DT for rectal surgeons have ranged from 3 to 45 min (P=0.004), SCT have range from 15 to 470 min (P=0.012) and TOT have ranged from 215 to 720 min (P=0.005).

Fifty-five RALH were performed by 2 gynecological surgeons, 28 procedures were done by Surgeon 4 and 27 procedures by Surgeon 5. There were 2 (3.5%) conversions to open surgery related to advanced disease, one for each surgeon. DT in this group has ranged from 2 to 35 min, SCT have ranged from 51 to 334 min and TOT have ranged from 105 to 430 min. 58 RALRH were performed by the same gynecologists. Surgeon 4 performed 33 procedures and Surgeon 5 performed 25 procedures. There was no conversion to open surgery in this group. DT in this group have ranged from 3 to 31 min, SCT have ranged from 74 to 346 min and TOT have ranged from 123 to 417 min. Overall, gynecological surgeons have operated 113 cases, Surgeon 4 operated 61 cases and Surgeon 5 operated 52 cases.

One hundred and twenty seven RALRP were performed by 2 urologists, 53 surgeries by Surgeon 6 and 74 by Surgeon 7. Surgeon 7 had one conversion (1.3%) related to technical difficulty during his second phase of the LC. DT in this group has ranged from from 1 to 81 min, SCT have ranged from 75 to 480 min and TOT has ranged from 151 to 514 min.

Linear regression graphs of the operative times arranged in chronological order are shown in Figures 1-3. Neither surgeon had a profound decrease of the operative times in the linear regression analysis. Notwithstanding, the majority of the surgeons were able to show a slightly and continuous improvement of the operative times over the analyzed learning period. Once all operative times were arranged, CUSUM values for each of the cases was calculated and the generated charts are shown in Figures 4-6. For most surgeons, the CUSUM graphs exhibited 3 LC phases: phase 1 identified by the first peak point, phase 2 identified by a stable line and phase 3 identified by the last peak point followed by significant slop in the curve. The CUSUM turning point reflecting the competency and proficiency for each surgeon regarding the different operative steps is summarized in Table 2.

Figure 1 Linear regression chart for docking time (DT).

Figure 2 Linear regression chart for surgeon console time (SCT).

Figure 3 Linear regression chart for total operative time (TOT).

Figure 4 Cumulative sum (CUSUM) chart for docking time (DT).

Figure 5 Cumulative sum (CUSUM) chart for surgeon console time (SCT).

Figure 6 Cumulative sum (CUSUM) chart for total operative time (TOT).

CUSUM graph for DT shows that all surgeons were able to achieve competency and proficiency with three distinct phases in this specific robotic surgical step, exception for RALRP Surgeon 6 who did not have proper curve since he already started with low DT and had few variances over the period. Competency and proficiency for RALTME surgeons could not be determined for SCT and TOT since no proper LC was stablished on the CUSUM graph. This trend reflects the wide operative time variance observed in the linear regression charts. Surgeon 5 also had wide variances of SCT for RALH on the CUSUM analysis for and no proper LC could be described. Despite, he had better overall SCT performance when comparing to RALH Surgeon 4. Urologist exhibited the more defined competency and proficiency turning point of the SCT and TOT CUSUM curves, followed by the gynecologists.

To determine the effect of BMI on the LC, a linear regression analysis pooling all 343 surgeries regarding DT, SCT and TOT were made and the graphs are plotted in Figure 7. Comparison of patient’s BMI and operative data are shown in Table 3. There was a clear inverse correlation between BMI and DT. Patients with higher BMI had a shorter DT and patients with lower BMI showed increased DT. For this surgical step, the increase of BMI seemed to help surgeons to dock the cart accordingly with Spearman’s rank correlation coefficient (P=0.016 and R=0.024). On the other hand, no association could be made between BMI and SCT or TOT (P=0.319 and P=0.857, respectively). When divided into two groups, the previous assumption was also confirmed. Patients with BMI <25 kg/m² had higher median DT than patients with BMI ≥25 kg/m² with statistical significance (P=0.041). Comparison of EBL over the learning period did not show any increase or decrease trend after linear regression analysis for any of the surgeons and no statistical correlation could be made, therefore this data could not be used as a parameter for LC correlations. No chart was drawn for this quality indicator. Nevertheless, correlation of EBL with BMI showed a higher median blood loss among patients with BMI <25 kg/m² (P=0.020).

Figure 7 Linear regression chart quantifying the associations between body mass index (BMI) and robotic-assisted laparoscopic surgery (RAS) operative times.

Discussion

The development of robotic platforms with its 3-D camera and fully articulated robotically driven instrumentation has enabled surgeons to perform minimally invasive surgery on the pelvis much like an open procedure. These advantages might facilitate the surgeons to advance effortlessly along the learning phase and previous publication analyzing proficiency development shows that robotic surgery has a fast and robust surgical skill acquisition when compared to other minimally invasive methods (25-28).

The LC can be described by a graphic representation of the temporal relationship between the surgeon’s mastery of a specifically surgical procedure and the chronological number of cases performed. The CUSUM method has been used as an indicator of satisfactory outcomes in relation to the acquisition of a surgical skill. The main advantages of this approach are the independence from sample size, effectiveness in detecting small shifts in patterns, and the ability to allow continuous analysis in time and rapid evaluation of data (15-19). Comparison of the robotic LC between different surgical specialties by this method has not yet been performed. Such study is crucial in order to evaluate if a specific robotic training program is necessary to attend the different particularities of each surgical specialty of the pelvis. To our knowledge, this is the first study comparing different specialties in the learning process of RAS.

Here we report the LC of RAS in the surgical management of pelvic malignancies of a tertiary oncological referral center in Brazil. Using the CUSUM method, the LC of different surgeons of the Urology, Gynecology and Abdominopelvic Departments could be demonstrated. Analysis of our data suggests that LC for each respective robotic operative step, surgeon and specialty is unique. Most surgeons presented a proper LC for the different operative times with three distinct phases as described by other authors analyzing the robotic surgical skill acquisition (29-39). In our study, slightly and continuous improvement of operative times in the linear regression charts were observed for the majority of surgeons in all robotic surgical steps. Furthermore, the CUSUM charts show that all surgeons from the different specialties were able to develop competency and proficiency, each one with its own specific characteristic. Exception for these affirmations occurred for RALTME surgeons.

All RALTME surgeons presented a slight increase of SCT and TOT over the period as shown on raw linear regression charts and no proper LC could be observed in the CUSUM graphs. The first reason for the difficulty in obtaining a LC is the change in the surgical approach that RALTME surgeons performed during the learning period. In the early beginning of robotic experience all surgeons approached the mesenteric vessel and splenic flexure by traditional laparoscopic technique and then docked the robotic system only for the mesorectal dissection. As the experience grew, surgeons started to adopt single docking for a fully robotic surgery. This innovation in the technical approach might explain the longer SCT at the final phase of the study, since a more comprehensive part of the surgery was done by the robotic platform. Recent publications corroborate our finds by reporting that surgeons tend to include more challenging cases and difficult procedures toward the latter part of their training period (29,33,35). The second major reason is the simple fact that rectal surgery requires multi quadrant operation (which means to operate on the upper abdomen for splenic flexure mobilization, left lumbar and left iliac regions for left colon dissection and on the pelvis for mesorectal excision) making such robotic intervention more laborious, time consuming and more difficult to master than single quadrant robotic surgery.

The current analysis indicates that docking is an easy step to master. All surgeons achieved competency and proficiency with few cases. Median DT for RALTME surgeons ranged from 11 to 18 min. Other authors have found similar median DT ranging from 4 to 12 min (30,32-34,36). However, Jiménez-Rodríguez et al. in a study analyzing 43 patients undergoing RAS for rectal cancer have found a mean DT of 62.9 (±24.6) min (29), but this is an isolated. All these finds demonstrate that DT can widely diverge between different surgeons, specialties and studies, but most importantly, this specific step can be mastered over the cases.

The CR of a minimally invasive procedure reflects its technical complexity. This affirmation is reinforced by reported CR as high as 22% for laparoscopic surgery for rectal, prostate and gynecological cancer (36-38,40-43). Meanwhile, previous publication addressing RAS for rectal, endometrial, cervical and prostate cancer found low conversion to open surgery. For robotic prostatectomies, authors have described a CR ranging from for 0% to 1.17% (40,44-46). The CR for RALRP in our study was 0% for RALRP Surgeon 6 and 1.3% for RALRP Surgeon 7. Published studies analyzing the CR of RALTME show a relatively low rate ranging from 0% to 3.2% (27,31,32,34,35,40,47-52). In our series, CR ranged from 0 to 7.3%. Previous reports regarding CR of RAS in gynecological oncology range from 0% up to 12.4% (38,43,53-56). In our series the CR for RALH and RALRH was 3.5% and 0%, respectively. It is believed that reduction of EBL over the learning period can also reflect the LC, however this trend was not observed in our study. We identified a low and stable rate of operative bleeding over the period for all surgeons. Correlation of EBL with BMI showed a higher median blood loss among patients with low BMI with significant difference (P=0.020). However, this outcome must be interpreted with caution since it was not the primary endpoint of our study. Our results regarding CR and EBL, in addition to the previous publications, recognize the safety of RAS in the learning period phase.

Even though it was not our primary objective, our analysis demonstrates correlation between DT and BMI. Contrarily of our primary assumption, based on previous reports comparing BMI and surgical outcomes (57), we observed a slight slope in DT as patient’s BMI increased on the linear regression chart (P=0.016). Same correlation was observed when patients were divided into two groups and compared by two sample means statistical test (P=0.041). We infer, based on our results, that obese patients supposedly have a wider abdominal surface comparing to thin patients and that it could give more freedom to surgeons when setting the robotic ports and arms, making this surgical step easier in this group of patients. Notwithstanding, no correlations regarding BMI and SCT or TOT could be made.

Training and proctoring are fundamental aspects in the acquisition of a new surgical skill. All the three surgical specialties had the same proctor and learning program, however urologists could develop the most consistent LC among all groups. Gynecological surgeons also had a more delineated LC for DT, SCT and TOT compered to rectal surgeons. It is noteworthy that gynecologists have operated an overall greater number of RAS compared to rectal surgeons. The acquired learning skills from one procedure are interchangeable between them since most of the robotic surgical steps are similar during RALH and RALRH. Therefore, gynecologists might have an advantage in the learning process compared to rectal surgeons. What cannot be determined from our analysis is whether the lack of a proper LC for rectal surgeons reflects the absence of a learning process or if changes in the surgical approach during the learning phase could have biased the outcomes. We believe that rectal surgeons have to master a more difficult docking step (they have to set the robotic arms in such way so they can reach the upper and the low abdomen) and also a more laborious surgeon console step, since a wider surgical field has to be approached. Meanwhile, urologists and gynecologists work only on the pelvis, which is a more restricted surgical field. For this reason, we believe that both the LC of docking and the surgeon’s console can be more quickly mastered by these specialties.

Rectal surgeons had greater difficulty in achieving competence and proficiency with less than 50 cases. Previous reports have shown that the LC for robotic assisted colorectal surgery can be achieved after 15 to 50 cases (29,30,33,34,58). However, only Odermatt et al. analyzed proctoring in the LC and hypothesized that high number of proctored cases may have a considerable impact on the LC (58). Possibly, if the RALTME surgeons in our study have had a longer training program in the simulator and had a greater number of cases proctored by an experienced robotic surgeon, the LC could have resembled the results of the other specialties analyzed here as well as previously reported LC for rectal RAS. Bowen et al. compared 44 patients undergoing pediatric robotic-assisted laparoscopic pyeloplasty and sought to determine if the LC would be affected by proctoring. They demonstrate that proctored surgeons could attain levels of expertise more quickly than those not proctored (59). Training and proctoring seemed to be the most important aspect of the LC. In consequence, intensive practicing and guidance program should be recommended for more laborious surgeries such as RALTME. Moreover, a different and more diligent training process might be necessary for RAS rectal surgery.

Comparison between different surgical specialties can be arguable. Every surgery has its peculiarities and the characteristics of each pathology are completely different. However, we are not analyzing surgical outcomes and raw operative times. In fact, we are comparing the capacity and speed of each surgeon and specialty developed the LC of RAS. Our analysis was not intended to measure absolute numbers, but to validate if there is a learning process and how different specialties and surgeons can master the technique. The CUSUM method and linear regression allowed an examination focused on trends and tendencies rather than raw numbers, ensuring a converged analysis on the learning process. Here we found the all the three surgeons in the RALTME had similar adversity to establish the LC. On the other hand, gynecologists and urologists exhibited similar trends inside each group representing well defined LC. Considering that LCs were homogeneous within each group, we can infer that our study reflects the real tendency in the development of robotic surgical skills of each specialty and not just of each surgeon.

Although we analyzed the LC of an expressive number of RAS by previous validated statistical methods, there are several limitations in our study. First, our study is a comparative retrospective analysis. Second, patients inside each group were not stratified by oncological stages. Third, outcomes were not evaluated as a LC parameter. Fourth, our data only describes a single institution experience. Yet, the strengths include prospective data collection, robust statistical analysis and enrollment of seven surgeons from three different specialties which better represent the robotic surgery learning process of a whole institution.

Conclusions

The present study shows that the LC for RAS in the management of pelvic malignancies does exist and is different and unique for each respective robotic operative step, surgeon and specialty, even though very similar among surgeons of the same specialties. Urological and gynecological RAS might have a less steep LC compared to RAS for rectal cancer. Therefore, robotic proctoring and training for rectal cancer should be more diligent. The validation of a new learned surgical skill should not be limited to the speed of operation, since it also involves other aspects such as gaining confidence and capability to manage a whole new spectrum of surgical environments and situations. Thus, a prospective multicenter study on different methods of LC analysis is necessary to validate our results.

Acknowledgments

We thank our colleagues from the National Cancer Institute (INCA), Federal University of the Rio de Janeiro State (UNIRIO) and Oswaldo Cruz Foundation (FIOCRUZ) who provided insight and expertise that greatly assisted the research. We thank Arnaldo Couto from Oswaldo Cruz Foundation for assistance with statistical analysis and Cumulative Sum calculations and Daiane Spitz from State University of Rio de Janeiro (UERJ) for comments that greatly improved the manuscript. We would also like to show our gratitude to Jose Alexandre Pedrosa, Urologist of the National Cancer Institute for sharing their pearls of wisdom with us during the course of this research, and we thank 3 “anonymous” reviewers for their so-called insights. We are also immensely grateful to Rodrigo Cesar and Deise Cesar for their comments on an earlier version of the manuscript, although any errors are our own and should not tarnish the reputations of these esteemed persons.

Funding: None.

Footnote

Conflicts of Interest: The authors have completed the ICMJE uniform disclosure form (available at http://dx.doi.org/10.21037/ls.2019.07.07). The authors have no conflicts of interest to declare.to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. The study was conducted in accordance with the Declaration of Helsinki (as revised in 2013). The study was conducted in accordance with the regulations of the local ethics committee and was approved by the institutional review board. All involved surgeons in this study signed the research informed consent. Patient informed consent was not necessary since it was a retrospective analysis, no intervention was performed and the subject of the study was the surgeon’s operative times.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. Heald RJ, Moran BJ, Ryall RD, et al. Rectal cancer: the Basingstoke experience of total mesorectal excision, 1978-1997. Arch Surg 1998;133:894-9. [Crossref] [PubMed]
2. Yaribakht S, Guillemin F, Harter V, et al. New approach of learning curve for robotic-assisted gynecologic oncology surgery. Gynecol Obstet Fertil 2015;43:348-55. [Crossref] [PubMed]
3. Davis JW, Shah JB, Achim M. Robot-assisted extended pelvic lymph node dissection (PLND) at the time of radical prostatectomy (RP): a video-based illustration of technique, results, and unmet patient selection needs. BJU Int 2011;108:993-8. [Crossref] [PubMed]
4. Stein JP, Lieskovsky G, Cote R, et al. Radical cystectomy in the treatment of invasive bladder cancer: long-term results in 1,054 patients. J Clin Oncol 2001;19:666-75. [Crossref] [PubMed]
5. Novara G, Ficarra V, Mocellin S, Ahlering TE, et al. Systematic review and meta-analysis of studies reporting oncologic outcome after robot-assisted radical prostatectomy. Eur Urol 2012;62:382-404. [Crossref] [PubMed]
6. Paley PJ, Veljovich DS, Shah CA, et al. Surgical outcomes in gynecologic oncology in the era of robotics: analysis of first 1000 cases. Am. J. Obstet. Gynecol 2011;204:551.e1-9. [Crossref] [PubMed]
7. ElSahwi KS, Hooper C, De Leon MC, et al. Comparison between 155 cases of robotic vs. 150 cases of open surgical staging for endometrial cancer. Gynecol Oncol 2012;124:260-4. [Crossref] [PubMed]
8. de Jesus JP, Valadão M, de Castro Araujo RO, et al. The circumferential resection margins status: A comparison of robotic, laparoscopic and open total mesorectal excision for mid and low rectal cancer. Eur J Surg Oncol 2016;42:808-12. [Crossref] [PubMed]
9. Lujan J, Valero G, Hernandez Q, et al. Randomized clinical trial comparing laparoscopic and open surgery in patients with rectal cancer. Br J Surg 2009;96:982-9. [Crossref] [PubMed]
10. Ohuchida K, Hashizume M. Robotic surgery for cancer. Cancer J 2013;19:130-2. [Crossref] [PubMed]
11. Koopmann MC, Heise CP. Laparoscopic and minimally invasive resection of malignant colorectal disease. Surg Clin North Am 2008;88:1047-72. [Crossref] [PubMed]
12. Mirnezami AH, Mirnezami R, Venkatasubramaniam AK, et al. Robotic colorectal surgery: type or new hope? A systematic review of robotics in colorectal surgery. Colorectal Dis 2010;12:1084-93. [Crossref] [PubMed]
13. Talebpour M, Alijani A, Hanna GB, et al. Proficiency-gain curve for an advanced laparoscopic procedure defined by observation clinical human reliability assessment (OCHRA). Surg Endosc 2009;23:869-75. [Crossref] [PubMed]
14. Page ES. Continuous inspection schemes. Biometrika 1954;41:100-15. [Crossref]
15. Wohl H. The cusum plot: its utility in the analysis of clinical data. N Engl J Med 1977;296:1044-5. [Crossref] [PubMed]
16. Williams SM, Parry BR, Schulp MMT. Quality control: an application of the CUSUM. BMJ 1992;304:1359-61. [Crossref] [PubMed]
17. Bolsin S, Colson M. The use of the Cusum technique in the assessment of trainee competence in new procedures. Int J Qual Health Care 2000;12:433-8. [Crossref] [PubMed]
18. Steiner SH, Cook RJ, Farewell VT. Risk-adjusted monitoring of binary surgical outcomes. Med Decis Making 2001;21:163-9. [Crossref] [PubMed]
19. Attilakos G, Sibanda T, Draycott T. Was The Higher Failure Rate Of The Kiwi Omnicup In A Randomized Study Attributed To A Learning Curve? Journal of Obstetrics & Gynaecology 2005;25:S24-S26.
20. Chaput de Saintonge DM, Vere DW. Why don’t doctors use CUSUMs? Lancet 1974;1:120-1. [Crossref] [PubMed]
21. Siegmund D. Sequential Analysis, Tests and Confidence Intervals. New York: Springer 1974, p. 24-30.
22. Buchs NC, Pugin F, Bucher P, et al. Learning curve for robot-assisted roux-en-y gastric bypass. Surg Endosc 2012;26:1116-21. [Crossref] [PubMed]
23. Biau DJ, Resche-Rigon M, Godiris-Petit G, et al. Quality control of surgical and interventional procedures: a review of the CUSUM. Qual Saf Health Care 2007;16:203-7. [Crossref] [PubMed]
24. World Health Organization. Physical status: the use and interpretation of anthropometry. Report of a WHO Expert Committee. World Health Organ Tech Rep Ser 1995;854:1-452. [PubMed]
25. Corcione F, Esposito C, Cuccurullo D, et al. Advantages and limits of robot-assisted laparoscopic surgery: preliminary experience. Surg Endosc 2005;19:117-9. [Crossref] [PubMed]
26. Delaney CP, Lynch AC, Senagore AJ, et al. Comparison of robotically performed and traditional laparoscopic colorectal surgery. Dis Colon Rectum 2003;46:1633-9. [Crossref] [PubMed]
27. Melich G, Hong YK, Kim J, et al. Simultaneous development of laparoscopy and robotics provides acceptable perioperative outcomes and shows robotics to have a faster learning curve and to be overall faster in rectal cancer surgery: analysis of novice MIS surgeon learning curves. Surg Endosc 2015;29:558-68. [Crossref] [PubMed]
28. Moore LJ, Mark R, Wilson MR, et al. Robotic technology results in faster and more robust surgical skill acquisition than traditional laparoscopy. J Robot Surg 2015;9:67-73. [Crossref] [PubMed]
29. Jiménez-Rodríguez RM, Díaz-Pavón JM, Juan F, et al. Learning curve for robotic-assisted laparoscopic rectal cancer surgery. Int J Colorectal Dis 2013;28:815-21. [Crossref] [PubMed]
30. Foo CC, Law WL. The learning curve of robotic-assisted low rectal resection of a novice rectal surgeon. World J Surg 2016;40:456-62. [Crossref] [PubMed]
31. Park EJ, Kim CW, Cho MS, et al. Is the learning curve of robotic low anterior resection shorter than laparoscopic low anterior resection for rectal cancer? A comparative analysis of clinicopathologic outcomes between robotic and laparoscopic surgeries. Medicine 2014;93:e109 [Crossref] [PubMed]
32. Park EJ, Kim CW, Cho MS, et al. Multidimensional analyses of the learning curve of robotic low anterior resection for rectal cancer: 3-phase learning process comparison. Surg Endosc 2014;28:2821-31. [Crossref] [PubMed]
33. Bokhari MB, Patel CB, Ramos-Valadez DI, et al. Learning curve for robotic-assisted laparoscopic colorectal surgery. Surg Endosc 2011;25:855-60. [Crossref] [PubMed]
34. Yamaguchi T, Kinugasa Y, Shiomi A, et al. Learning curve for robotic-assisted surgery for rectal cancer: use of the cumulative sum method. Surg Endosc 2015;29:1679-85. [Crossref] [PubMed]
35. Sng KK, Hara M, Shin JW, et al. The multiphasic learning curve for robot-assisted rectal surgery. Surg Endosc 2013;27:3297-307. [Crossref] [PubMed]
36. El Hachem L, Momeni M, Friedman K, et al. Safety, feasibility and learning curve of robotic single-site surgery in gynecology. Int J Med Robot 2016;12:509-16. [Crossref] [PubMed]
37. Torng PL, Pan SP, Hwang JS, et al. Learning curve in concurrent application of laparoscopic and robotic-assisted hysterectomy with lymphadenectomy in endometrial cancer. Taiwan J Obstet Gynecol 2017;56:781-7. [Crossref] [PubMed]
38. Lim PC, Kang E, Park DH. Learning Curve and Surgical Outcome for Robotic-Assisted Hysterectomy with Lymphadenectomy: Case-Matched Controlled Comparison with Laparoscopy and Laparotomy for Treatment of Endometrial Cancer. J Minim Invasive Gynecol 2010;17:739-48. [Crossref] [PubMed]
39. Dal Moro F, Secco S, Valotto C, et al. Specific learning curve for port placement and docking of da Vinci(®) Surgical System: one surgeon’s experience in robotic-assisted radical prostatectomy. J Robot Surg 2012;6:323-7. [Crossref] [PubMed]
40. Kim NK, Kang J. Optimal total mesorectal excision for rectal cancer: the role of robotic surgery from an expert’s view. J Korean Soc Coloproctol 2010;26:377-87. [Crossref] [PubMed]
41. Kwak JM, Kim SH, Kim J, et al. Robotic vs laparoscopic resection of rectal cancer: short-term outcomes of a case-control study. Dis Colon Rectum. 2011;54:151-6. [Crossref] [PubMed]
42. Mak TWC, Lee JF, Futaba K, et al. Robotic surgery for rectal cancer: a systematic review of current practice. World J Gastrointest Oncol. 2014;6:184-93. [Crossref] [PubMed]
43. Lowe MP, Johnson PR, Kamelle SA, et al. A multiinstitutional experience with robotic-assisted hysterectomy with staging for endometrial cancer. Obstet Gynecol 2009;114:236-43. [Crossref] [PubMed]
44. Rocha R, Fiorelli RK, Buogo G, et al. Robotic-assisted laparoscopic prostatectomy (RALP): a new way to training. J Robot Surg 2016;10:19-25. [Crossref] [PubMed]
45. Tobias-Machado M, Mitre AI, Rubinstein M, et al. Robotic-assisted radical prostatectomy learning curve for experienced laparoscopic surgeons: does it really exist? Int Braz J Urol 2016;42:83-9. [Crossref] [PubMed]
46. Vasdev N, Bishop C, Kass-Iliyya A, et al. Developing a robotic prostatectomy service and a robotic fellowship programme - Defining the learning curve. Curr Urol 2013;7:136-44. [Crossref] [PubMed]
47. Papanikolaou IG. Robotic Surgery for Colorectal Cancer Systematic Review of the Literature Surg Laparosc Endosc Percutan Tech 2014;24:478-83. [PubMed]
48. Staderini F, Foppa C, Minuzzo A, et al. Robotic rectal surgery: state of the art. World J Gastrointest Oncol 2016;8:757-71. [Crossref] [PubMed]
49. Park EJ, Cho MS, Baek SJ, et al. Long-term oncologic outcomes of robotic low anterior resection for rectal cancer: a comparative study with laparoscopic surgery. Ann Surg 2015;261:129-37. [Crossref] [PubMed]
50. Patriti A, Ceccarelli G, Bartoli A, et al. Short- and medium-term outcome of robot-assisted and traditional laparoscopic rectal resection. JSLS 2009;13:176-83. [PubMed]
51. Baik SH, Kwon HY, Kim JS, et al. Robotic versus laparoscopic low anterior resection of rectal cancer: short-term outcome of a prospective comparative study. Ann Surg Oncol 2009;16:1480-7. [Crossref] [PubMed]
52. Xiong B, Ma L, Zhang C, Cheng Y. Robotic versus laparoscopic total mesorectal excision for rectal cancer: a meta-analysis. J Surg Res 2014;188:404-14. [Crossref] [PubMed]
53. Favre A, Huberlant S, Carbonnel M, et al. Pedagogic Approach in the Surgical Learning: The First Period of “Assistant Surgeon” May Improve the Learning Curve for Laparoscopic Robotic- Assisted Hysterectomy. Front Surg 2016;3:58. [Crossref] [PubMed]
54. Holloway RW, Ahmad S, DeNardis SA, et al. Robotic-assisted laparoscopic hysterectomy and lymphadenectomy for endometrial cancer: Analysis of surgical performance. Gynecol Oncol 2009;115:447-52. [Crossref] [PubMed]
55. Payne TN, Dauterive FR. A comparison of total laparoscopic hysterectomy to robotically assisted hysterectomy: surgical outcomes in a community practice. J Minim Invasive Gynecol 2008;15:286-91. [Crossref] [PubMed]
56. Seamon LG, Cohn DE, Richardson DL, et al. Robotic hysterectomy and pelvic-aortic lymphadenectomy for endometrial cancer. Obstet Gynecol 2008;112:1207-13. [Crossref] [PubMed]
57. Herman MP, Raman JD, Dong S, et al. Increasing body mass index negatively impacts outcomes following robotic radical prostatectomy. JSLS 2007;11:438-42. [PubMed]
58. Odermatt M, Ahmed J, Panteleimonitis S, et al. Prior experience in laparoscopic rectal surgery can minimize the learning curve for robotic rectal resections: a cumulative sum analysis. Surg Endosc 2017;31:4067-76. [Crossref] [PubMed]
59. Bowen DK, Lindgren BW, Cheng EY, et al. Can proctoring affect the learning curve of robotic-assisted laparoscopic pyeloplasty? Experience at a high-volume pediatric robotic surgery center. J Robot Surg 2017;11:63-7. [Crossref] [PubMed]