Surgical Activity Recognition

In our paper analyzing surgical activity within surgical tasks [LINK] we outline a hierarchical decomposition of the intraoperative surgical process from the surgeon’s perspective. We proposed four levels of surgical activity: procedure, phase / task, maneuver, and gesture. For example, the figure below shows the decomposition of a hysterectomy procedure. The procedure goes through phases like ligation (dissection of ligaments), bladder dissection,lymph node dissection, colpotomy, and vaginal cuff closure (suturing). Within vaginal cuff closure, a series of maneuvers need to be performed, e.g., needle passing, running the suture, tying knots, cutting extra suture. Furthermore, each maneuver is composed of a sequence of gestures like grasping the needle, positioning the needle, driving it through the tissue, pulling it out, and so on.

We have developed algorithms based on machine learning methods that can analyze surgical video images and/or instrument motion data to predict what activity is performed at a given time instant during the surgery. According to the granularity of the activity being predicted we term the problem as phase recognition, maneuver recognition, and gesture recognition.

  1. Vedula SS, Malpani AO, Tao L, Chen G, Gao Y, Poddar P, Ahmidi N, Paxton C, Vidal R, Khudanpur S, Hager GD, Chen CCG. Analysis of the structure of surgical activity for a suturing and knot-tying task. PLOS ONE. 2016 Mar 7;11(3):e0149174.
  2. DiPietro R, Ahmidi N, Malpani A, Waldram M, Lee GI, Lee MR, Vedula SS, Hager GD. Segmenting and classifying activities in robot-assisted surgery with recurrent neural networks. Int J Comput Assist Radiol Surg. 2019 Apr 29; PMID: 31037493

Technical Skill Assessment

Surgical technical skill is currently measured using validated global rating scales (GRS) that may be catered to a form of surgical intervention (endoscopic vs. open surgery) or developed for a specific procedure (e.g. cataract surgery). For example, GEARS [LINK] was developed to assess generic robot-assisted surgery performance, while OSCAR [LINK] was developed to assess technical skill in cataract procedures. At times, there is no established method like GRS to assess the surgical skill. In such cases, experience of surgeon or their level of training is used as a proxy for their surgical skill. While this is not the best estimator for skill, we use it if GRS are lacking.

We have crafted methods and applied machine learning techniques on instrument motion and video images to analyze surgical technical skill. We have done this in two formulations. First, classifying a performance into categories like novice, intermediate, and expert which we term as skill classification. Second, estimating a skill score for the performance which we term as skill scoring.

Our review article [LINK] describes existing works on automated objective skill assessment in surgical context.

  1. Vedula SS, Ishii M, Hager GD. Objective Assessment of Surgical Technical Skill and Competency in the Operating Room. Annual Review of Biomedical Engineering. 2017;19(1):301–325. PMID: 28375649
  2. Ahmidi N, Poddar P, Jones JD, Vedula SS, Ishii L, Hager GD, Ishii M. Automated objective surgical skill assessment in the operating room from unstructured tool motion in septoplasty. Int J CARS. 2015 Apr 17;1–11.
  3. Malpani A, Vedula SS, Chen CCG, Hager GD. A study of crowdsourced segment-level surgical skill assessment using pairwise rankings. Int J CARS. 2015 Jun 30;10(9):1435–1447.
  4. Ahmidi N, Gao Y, Bejar B, Vedula SS, Khudanpur S, Vidal R, Hager GD. String Motif-Based Description of Tool Motion for Detecting Skill and Gestures in Robotic Surgery. In: Mori K, Sakuma I, Sato Y, Barillot C, Navab N, editors. Medical Image Computing and Computer-Assisted Intervention – MICCAI 2013 [Internet]. Springer Berlin Heidelberg; 2013 [cited 2013 Nov 22]. p. 26–33. PMID: 24505645

Autonomous Surgical Systems

More details to come soon.