How Computer Vision and Big Data Analytics are Revolutionizing Surgery

Alaa El-Hussuna¹, Luca Gordini^2
1Coordinator OpenSourceResearch Collaboration,
²Chirurgo Generale presso Gruppo San Donato

Computer vision in surgery

Forced turnovers, number of attempts on target, possession control, ball recovery time, line breaks, defensive line height and team length, receptions behind midfield and other football metrics provide big data. Mining of this big data enabled team managers to plan and change strategies.

Already, the adoption of analytics by most elite teams means the advantage conferred has shrunk. The launch of giant player databases has aided due diligence on potential signings. Tactics have changed too: long-range shots and crosses have declined in the Premier League as data has shown they might lead to fewer goals than many coaches realised (1).

If big data can change football team success, it can certainly change surgical team performance. That is what OpenSourceResearch Collaboration (OSRC) is exploring through a target-oriented research program.

                    Computer vision research to improve surgical skills

OSRC is an international organisation focused on implementing information technologies in medical research and innovation. The organisation “working model” cherishes the multi-disciplinary approach to solving challenges in traditional research. Teams are usually composed of experienced researchers, clinicians, data and computer scientists. The organisation has published 21 articles in leading journals and conducted research workshops in developing countries (5).

Computer vision has a wide range of applications in the field of surgery which may lead to improve surgical outcomes, enhance patient safety, reduce complications, and assist surgeons in delivering more precise and efficient care.

When it comes to surgical films, computer vision can be utilized in various ways to extract valuable information and enhance surgical education, research, and quality assessment. Here are some ways in which computer vision can be applied to surgical films (2,3):

1. 1. Surgical activity recognition: Computer vision algorithms can analyse surgical videos to automatically recognize and classify different surgical activities and gestures. By identifying specific actions such as suturing, tissue dissection, or cauterization, computer vision can provide valuable insights into the surgical workflow, enabling efficient video indexing and retrieval.
   2. Surgical skill assessment: Computer vision can analyse surgical videos to assess the technical skills of surgeons. By tracking and analysing the movements of instruments, hand-eye coordination, and overall surgical performance, computer vision algorithms can provide objective measures of skill proficiency. This information can be used for training, feedback, and quality improvement.
   3. Surgical tool detection and tracking: Computer vision techniques can detect and track surgical instruments and tools in surgical videos. This can assist in understanding the usage of different tools during the procedure, identifying instrument exchanges, and tracking the tool’s trajectory. Such information can aid in surgical workflow analysis and instrument recognition.
   4. Gesture recognition and interaction: Computer vision can be used to recognize and interpret hand gestures or surgeon-patient interactions in surgical videos. This can enable the extraction of information related to communication, teamwork, and ergonomics within the surgical team, facilitating studies on human factors and team dynamics.
   5. Anomaly detection: Computer vision algorithms can analyze surgical videos to identify unexpected or abnormal events during the procedure. This can include detecting complications, deviations from the standard surgical protocol, or unusual anatomical variations. Anomaly detection in surgical films can help with improving patient safety, providing feedback for error prevention, and contributing to research on adverse events.
   6. Surgical workflow analysis: By analyzing surgical videos, computer vision can provide insights into the sequence of surgical steps and the overall workflow. This can help with identifying areas for process improvement, standardization of procedures, and optimization of surgical techniques.
   7. Surgical education and training: Computer vision can enhance surgical education by automatically indexing, annotating, and summarizing surgical videos. It can assist in creating educational content, developing virtual simulators, and providing personalized feedback to trainees. Additionally, computer vision techniques can be used to compare surgical techniques across different videos or identify exemplar videos for educational purposes.

Data analysis provides surgeons with feedback that can be used to improve performance and will shorten training time needed to achieve peak performance in surgery

By leveraging computer vision techniques on surgical films, we can gain valuable information, facilitate research, improve surgical training, and enhance quality assessment in the field of surgery.

Let us examine the implementation of computer vision in surgery in more detail. Zooming in we shall discover two interrelated important concepts widely used to analyse surgical video films (2); automated performance measures and surgical phase recognition.

Automated performance measures in surgery

Automated performance measures using computer vision can provide objective and quantitative assessments of surgical skills and performance (4). These measures can be valuable for surgical training, evaluation, and quality improvement. Here are some examples of automated performance measures in surgery:

1. Tool usage and movement analysis: Computer vision algorithms can track and analyze the movements of surgical instruments during a procedure. By measuring metrics such as instrument path length, speed, and smoothness, computer vision can provide quantitative insights into the efficiency and dexterity of surgeons. It can identify deviations from optimal movement patterns and provide feedback on instrument handling.
2. Time analysis: Computer vision can automatically measure the duration of specific surgical steps or the overall procedure. This allows for the evaluation of efficiency, standardization, and comparison of surgical techniques. It can also assist in identifying time-consuming tasks that may need improvement or optimization.
3. Gesture recognition and economy of motion: Computer vision can recognize and analyze hand gestures and movements of surgeons. By assessing factors such as the number of unnecessary or extraneous movements, economy of motion, and ergonomics, computer vision can provide feedback on the efficiency and ergonomics of surgical performance.
4. Accuracy and precision: Computer vision can assess the accuracy and precision of surgical actions. For example, it can measure the distance between a surgeon’s tooltip and a target location, or evaluate the accuracy of suturing or knot tying. These metrics can allow identifying areas for improvement and provide feedback on precision-based surgical skills.
5. Proficiency in tissue manipulation: Computer vision algorithms can analyse the interaction between surgical instruments and tissue. They can assess factors such as tissue handling, traction, and the forces applied during tissue manipulation. These measures can help evaluate the surgeon’s proficiency in delicate tissue handling and minimize the risk of tissue trauma.
6. Procedural adherence: Computer vision can compare surgical actions to established guidelines or standard procedures. It can assess adherence to predefined steps, identify deviations or omissions, and provide feedback on the consistency of surgical technique. This can be particularly useful for ensuring compliance with best practices and safety protocols.
7. Cognitive workload assessment: Computer vision can analyse eye gaze patterns and pupil dilation to assess the cognitive workload of surgeons during a procedure. This information can provide insights into attentional focus, decision-making processes, and mental workload. It can help identify areas where the cognitive load may be high and support interventions to improve performance and reduce errors.

Thus, automated performance measures using computer vision can augment traditional subjective evaluations and provide objective, standardized, and quantitative feedback on surgical skills and performance. By analysing various aspects of surgical actions and techniques, these measures can contribute to surgical education, skill assessment, training, and quality control. Automated performance measures and surgical phase recognition are closely related in the context of computer vision applications in surgery as we shall see in the next section.

Surgical Phase Recognition Using Computer Vision

Surgical phase recognition refers to the ability of computer vision algorithms to identify and classify different phases or stages of a surgical procedure based on the analysis of visual cues in surgical videos or images (3).

How automated performance measures enable phase recognition

Automated performance measures can contribute to surgical phase recognition by providing quantitative metrics that help differentiate between different phases. By analysing the movements, actions, and characteristics of surgical instruments, as well as other visual cues, computer vision algorithms can determine when a transition occurs between different surgical phases.

For example, during a laparoscopic cholecystectomy (gallbladder removal), automated performance measures can analyse the movements and usage of surgical instruments to identify the incision phase, dissection phase, clipping and cutting phase, and so on. These performance measures, such as instrument path length, force applied, or gestures performed, can serve as discriminative features to classify and recognize the different phases of the procedure.

Benefits of combining performance measures and phase recognition

By combining automated performance measures with surgical phase recognition, computer vision systems can provide a comprehensive understanding of the surgical workflow. Surgeons, educators, and researchers can benefit from this information in various ways. It enables the analysis of surgical techniques and efficiency across different phases, the identification of critical events or transitions, the evaluation of adherence to surgical protocols, and the assessment of performance and skill proficiency within specific phases of the procedure.

Ultimately, the integration of automated performance measures and surgical phase recognition helps automate the analysis of surgical videos, enables objective assessments, facilitates surgical education and training, supports quality improvement efforts, and enhances our understanding of the surgical process as a whole.

Huge amount of data is collected daily in surgical departments all over the world. Almost all of this data is not used. OSRC is exploring data mining research in which this data combined with data from patients and surgery analysed to improve surgical outcomes.

Improving surgical performance

During the world football tournament in Qatar, we saw the use of Microsoft Analytics of football video films. This data analysis has helped team mangers to improve their team performance. If this data analysis helped in football, a game, why not using computer vision to improve surgeon performance? Performance in surgery might mean life and death.

References:

1. In the head, not on it, “Expected Goals” explains how data changed football. Data led to the rise of the “long-ball” game, then to its demise. The Economist Sep 1st 2022
2. Ward TM, Mascagni P, Ban Y, Rosman G, Padoy N, Meireles O, Hashimoto DA. Computer vision in surgery. Surgery. 2021 May;169(5):1253-1256. doi: 10.1016/j.surg.2020.10.039. Epub 2020 Dec 1.
3. Garrow CR, Kowalewski KF, Li L, Wagner M, Schmidt MW, Engelhardt S, Hashimoto DA, Kenngott HG, Bodenstedt S, Speidel S, Müller-Stich BP, Nickel F. Machine Learning for Surgical Phase Recognition: A Systematic Review. Ann Surg. 2021 Apr 1;273(4):684-693. doi: 10.1097/SLA.0000000000004425.
4. Eckhoff JA, Ban Y, Rosman G, Müller DT, Hashimoto DA, Witkowski E, Babic B, Rus D, Bruns C, Fuchs HF, Meireles O. TEsoNet: knowledge transfer in surgical phase recognition from laparoscopic sleeve gastrectomy to the laparoscopic part of Ivor-Lewis esophagectomy. Surg Endosc. 2023 May;37(5):4040-4053. doi: 10.1007/s00464-023-09971-2. Epub 2023 Mar 17. PMID: 36932188;
5. Open Source Research Organisation: Implementing Information Technologies In Medical Research (osrc.network)