How a Last-Minute Surgery Observation Led to a Breakthrough in Surgical Robotics

Yeah, it's a very interesting question, um. There is a, there is a project I recall from, um, from my research work in, in, in graduate school. Um, we were working with, um, this particular surgeon, um, Johns Hopkins Hospital, um, and our team. Had some ideas about how we could help the surgeon with his procedure by, you know, um, this is a surgical robotics, so the surgeon sees something on a screen, and we want to provide some added value in his surgery. Can we, can we add some overlays? Can we show some information that maybe otherwise he, he couldn't see, you know, just by looking at the camera inside the patient, and we, we had a particular idea about a novel technology. You know, kind of trying, asking the, the surgeon, he, he was supposed to use a tool and then we would show something on the screen about what's underneath the surface, right? What, what's in the tissue of the patient. And um we're very, very proud of, of the idea we had, so we invited this, this surgeon in our, in our lab to, to show him a demo. Well, we, of course, didn't have, you know, we had some props. We didn't have anything that really looked like what he was encountering in the operating room, and we didn't know anything at that time. And so students and professors, excited to meet the big surgeon, made a presentation, and the surgeon was very lukewarm. Well, it's kind of interesting, but I'm not really sure how am I gonna use this, and um he really kind of just wanted to leave, and I remember thinking that um we are missing something, right? We are missing something. So, so I, I said, you know, Doc Doctor Suu, I, I have an idea. Could we, would it be possible for one of us to come and see one of your surgeries? And he said, yes. So, um, you know, I mean, besides the fact that it was fascinating, um, we, I went, I went to the surgery room, and the thing that was cool, you know, I was trying to reconcile all this new information that I was getting, and I just remember at one point, the surgeon stopped in the middle of the operation, he was behind the console, and he said, come over here. All I need is this, in this 5 seconds, when I make this decision, that's when I need the information from your tool. I don't need it in the other 1.5 hours of the surgery, I only need it then. And I remember after the surgery coming back to the office, and actually he said, go and tell Professor, you know, X. That I love all his ideas, but that's when I need help. That's when I realized that, you know, we didn't have that information. We were thinking about this project as, you know, having unlimited time and unlimited resources and just being able to do all sorts of fancy things and, um, you know, of course I went back to the lab and we kind of retooled. Everything from our experiment, the prop we were using to show, um, you know, our technology, and I also felt like it was such a good example of how somehow, sometimes engineers get very Enamored with the technology they are, they are devising and there's sometimes, you know, kind of um lose sight of. When would it actually be applicable, when and how it could be used, so it was a, it was a really eye opening experience.

Well, that's really important, so we definitely thought about. You know, the time our experiment needed to run, the time our tool needed to produce results. So we wanted to have something that showed up really fast, we ended up implementing a multi-threading version of our algorithm. To make it work super fast, to be able to produce results right away. We had all sorts of annotations. We had these things that we call like haptic barriers. We basically implemented something so that when the surgeon is in the operating room, the tool would be prevented from going into areas that would be unsafe. Let's say there was like a blood vessel and he would hold a knife, right? So, um, so we, we really tried to, um. So change both the speed with which our algorithm was working and also um the usability of the things that we were overlaying on the, on the screen for, for the surgeon.

Um, so, we were using an ultrasound probe, and, um, one of the things that we considered was the idea of creating a very small 3D ultrasound. So somewhere where you do multiple sweeps and you kind of try to create a volume. And even though it seemed to make sense because you want the surgeon to have a more 3D picture and kind of see a volume, we were looking at detecting some very, very tiny nerves. Basically, imagine some very thin bands of fabric that was thicker than the rest. And so we realized that if we would go the 3D route. A, it would be more time consuming, so the algorithm would take longer to compute and present the results, and there is an inherent fault in a, in a 3D probe where as you go further away from the surface, your slices become further away from one another, so you're actually not presenting very trustworthy data at. Higher depth right from the surface and our focus point we found out that day from the sur from the surgeon was in that first like 1 millimeter behind the surface, so we discounted the 3D probe that we were considering um using, you know, can we be fancy, can we do 3D? Actually, no, in this case we don't need 3D. OK.