Total Product Lifecycle Security: From Design to Disposal | Ep. 27

Hello and welcome back to the Med Device Cyber Podcast. We are joined here by your co-host Christian Espinosa, and today we are going to be talking about Total Product Lifecycle Security and developing a secure software development life cycle. How are you doing today, Christian? I am doing good. This is one of my favorite topics. It is something that the TPLC, or the SPDF, is something that is commonly neglected and is often why software ends up unsecure or insecure in my opinion. But I am doing pretty good. I did put on some, not enough, it wasn't SPDF, it's uh, what do you call that? The S, the sunscreen. I had I put some of that on the other day at the beach, you know, SPF. SPF. Yeah, there you go. So, I still got a little bit sunburned. Um, but yeah, I'm stuck, uh, stuck in Florida for a couple days. Uh, worst place to be stuck, I guess. But, uh, you know, travel delays going through Dallas. I figured I would rather be stuck here than Dallas. So, yeah, that is what is going on with me. It seems like there is a delay anytime you fly through Dallas. I think they are just not used to weather at all there. So, if the wind shifts the wrong direction, they shut down the airport. Yeah. So, maybe I will avoid Dallas next time. But there you go. So, let us go back to TPLC and SPDF. Um, SPDF, what are those like the same, or are they really different? What do you think? So, TPLC and SPDF, I think that the Secure Product Development Framework is part of the Total Product Lifecycle. So, when we are looking at Total Product Lifecycle, total is kind of the key word there. It goes from the concept phase all the way to decommissioning whenever you are done supporting the product. So, it needs to cover everything. The Secure Product Development Framework, product development is an ongoing effort. It is not a one-and-done situation. It keeps going throughout the life of the product. So, it is a framework that ensures you are managing security at every step of the way. You are not missing any big considerations. Um, you are designing it with security at the front of mind, you are performing regular code checks, going through the design process, and you are not leaving security to just a time block at the end. Um, that I think the synonymous part is the SSDLC, the Secure Software Development Life Cycle, which goes into that cyclical process. So, you are making a change to it, you are reviewing the change, you are implementing the change, you are testing the change, and then you go in again, you are making another change. All of that needs to have security at the front of mind. There are a lot of processes, a lot of tools, lots that goes into a Secure Software Development Life Cycle. So, the SPDF, the Secure Product Development Framework is really synonymous with an SSDLC, a Secure Software Development Life Cycle. Would you say that? Just about. Yeah, they are pretty similar. And then all of that ties into the Total Product Life Cycle. So, it is a component of, you know, the full, the full product. Obviously, the development is the main part of the product. You have an initial device. You keep making tweaks and changes. You keep developing it. You keep changing it. And so, that is what we are looking at with the Total Product Life Cycle. And that is what we are looking at with the Secure Product Development Framework. Yeah. And I think the Total Product Life Cycle is something that needs more emphasis. And I think that is why it is a requirement. Now, I know in the past I have worked with a medical device manufacturer that had the assumption, which is a true assumption, that the device would be in a secure room in a hospital. But what they did not consider is when the device is decommissioned and the hospital no longer wants it, what were they going to do with the device? And these devices did not have encrypted hard drives. So, the hospitals were getting rid of these devices. People were able to purchase them off of eBay and other sources and grab all the PHI off the hard drive. So, they totally kind of forgot about that whole decommissioning and the security involved with that. And this even applied like back in the day when I worked for the government, the DoD, they would get rid of like printers. And these some of the printers were classified printers and they would just sell them to whoever wanted to buy them. And a lot of these printers had hard drives on them with classified documents. So, I I think it is extremely important to think about from like you said, concept to decommissioning, the security in that entire process because it is often forgot about once the product is sold. And that is even like there there is postmarket management which is once it is sold, but there is also the decommission aspect and both of those need to be considered for the TPLC. Yeah. Yeah. It all goes in all the way down the line. And a lot of times, yeah, manufacturers are not thinking about that. They say, “Well, when the device is done, it is done.” But if you throw that in a dumpster and it has patient records on it, um, I know now everyone is pushing to like cloud EHR systems. But before when EHR was coming into play, everything was on-prem. And then when you have these on-prem EHR systems, people just leave them on the computer and throw them out, and someone could go dumpster diving and then pull out thousands of patient records. And it seems like a bit of a silly way to hack the organization, but oftentimes the easiest way is the path of least resistance. Going into the dumpster is a lot easier than trying to hack from the outside into a hospital network. So, everything needs to be considered start to finish here. I think a really important part of it that is also often overlooked is what is the development environment like? So, not just the how you are building the product, not just the code going into it, but who is interacting with that code. How are you securing the code? How are you securing your workspaces? Are you just renting a WeWork office and leaving your laptop there when you go for lunch with an unlocked door? Anyone can make any changes that they want to it. Are you enabling multifactor authentication on your GitHub? A lot of this stuff manufacturers are not really thinking about and that can be a very very easy way for their device to be tampered with or compromised. Yeah, I think something that is also not considered very often and we have had a couple clients um have some challenges with this area is the update environment as well because when we are talking about the TPLC, we are talking about and the SPDF as well is if we need to patch our system because there is a vulnerability that came out, how do we securely develop that patch which is one part of it that is more the SPDF portion, but then how do we securely deploy that patch which is the other part and I think a lot of people just do not bother to consider that aspect of an attack. So, because of one of our clients, and I think you are familiar with this client, what the challenges were they they are able to do over-the-air updates to their products, but the environment they were doing the over-the-air updates from was basically their corporate environment. So, they are like engineers are writing the patches, right? We are connected to the same corporate network as HR and the accountants and other people that could easily click on phishing email and contaminate the engineer's laptop which could there then therefore contaminate all the products via the over-the-air update. So, I mean all these things need to be considered and it is really a lot to consider actually. Yeah, we had um kind of a similar situation where the client had a device and the device itself was extremely secure. We were helping test it, helping secure it, and we were just kind of drawing blanks trying to hack into it. There was not really an easy way to hack into this product. And what it was doing, it was taking in some information and then it was receiving updates. It was performing analysis on a local machine receiving updates from the cloud anytime there was a change and you can see you can capture that update process. It is encrypted, of course, because it is going over the open internet under HTTPS. You even look at the cipher suites. That is a common problem. So, what type of encryption are you using on that transfer? Everything was secure. And then we looked at the sort of the password for that transfer and it was pulled straight out of Stack Overflow with a publicly disclosed vulnerability in the uh key itself for the encryption for the data transfer. So, the encryption was secure, but the key was something that could be easily guessed and had been easily guessed before. It does not matter how great the encryption is if the key is very simple and you can break it, right? And so we were able to just guess the key and then we pushed out we could push out our own updates across all of the fielded devices with whatever we wanted on it by tampering what is going on in that update server. So, that is one of the vulnerabilities with uh I I think like there is always like the convenience factor of OTA over-the-air updates people like to say but there is also a major pathway for attackers like the scenario you just gave. If I can hack in the infrastructure now, I can push a malicious update to everybody very easily. Exactly. And you are not going to have that consideration if you have service techs take a USB thumb drive and upload it directly into the device, but then you have to deploy service technicians to all fielded locations. So, trade-offs on both sides. Um, are you more worried about convenience or likely easier security? But even having service techs go out, what if, you know, they go to the bathroom, someone swaps the USB drives, things like that. There always considerations there. Um, so there is never going to be perfect security. Yeah. Like I am at a hotel and I had to go print something. Um, I was supposed to like wet sign it like, you know, with a pen. So, I have a USB key and then there is a kiosk computer, but I was like, I am just going to like sign it electronically and see if they notice it is electronic versus a wet signature because I did not want to go and put this USB key in the kiosk computer. And if I am like, let us say I was the update engineer. I have got the the the update on here, but then I go use a kiosk in a hotel, that computer, which those are notoriously infected, you know, then now my update is going to be affected. So, you have to have a whole process around that as well to make it secure. Yep. Yeah. And so that goes into the Total Product Life Cycle. You have to have all of these processes covered. You have to touch up on every single area of security. You cannot leave any base uncovered. And so, you know, it goes outside of just the device in ways that a lot of manufacturers might not be thinking of. You know, who is doing these updates? Are is are you outsourcing it to contractors to do these updates? How can you trust these contractors? Who is doing your development? Is it done in-house? Do you know if, you know, you are outsourcing to a contract manufacturer? How are they developing this code? Are they doing it right? So, there are so many questions that can go into this and it is always going to depend on the device, the environment, how you are developing it, where you are developing it, but all of that needs to have an answer. It is almost like a tabletop exercise. You sit down and think of all the scenarios and the data flow for everything about your product and the how you update it, how you decommission it. you have to like think through like every use case basically to make make sure this is uh I guess totally covered right? Yeah, exactly. And it is something where we actually do that exercise, it is uh something that the FDA wants to see for medical devices not specifically for this, but that is threat modeling. And so, when we are doing threat modeling, of course, we are saying, “You know, what if someone uses a bad password on their device? What if there is bad encryption in the device? What if all these things go wrong?” And we are also saying, “What if developers do not have multifactor authentication on their GitHub accounts? What if their GitHub account password is password? What if they are leaving laptops open and working in a coffee shop and then they do not lock their laptop when they leave?” All of that goes into consideration. How are their controls that are going to prevent things from happening in the development environment, in the manufacturing environment, in the research environment? And so, threat modeling covers that whole Total Product Life Cycle outside of just looking at the device. And, you know, I am sure we have talked about threat modeling a lot and we have talked about all the common pitfalls, but that is a really big one with threat modeling is keeping the lens too narrow, focusing on the device instead of widening it and looking at the product and the systems involved. Yeah, and that is a good point. My cousin used to work for a major staffing firm. He was a software developer. He had all the information on a laptop. He left the laptop in his vehicle while he ran into a convenience store in the Houston area and someone stole his laptop. So, it was a major disclosure. There was like 80,000 client records on his laptop unencrypted. So, it is no different than a service tech going to update one of these devices. That laptop may have sensitive data. that USB key may have uh a patch on it, but how do we know that patch was not modified? Which goes back into that integrity and the hashing and maybe a digital signature to make sure the patch was not somehow modified. And then I think the other aspect that is often overlooked uh like when I did work in the DoD and I even commercial work a lot like you mentioned a lot of people move things to the cloud. A lot of softwares and medical devices deployed in the cloud. But the question I think people often neglect is where, I mean the cloud is the cloud, but physically in the cloud, like what what country is my data in? Because in the DoD, we we have had people in commercially too, they would be using a cloud, but they chose a site that is like in a country where what do you do if uh that country no longer likes the United States or there is a riot in the country or they someone breaks in the data center there and now your US government data only is in another country as example. So, I mean there is a lot of things to consider. Yeah. And even just considering where in the US is your data like a little bit of a fringe example but how are you if you are storing all of your information in a singular cloud location based in we will say Oklahoma or something or Florida. The US is prone to a lot of natural disasters. We get hurricanes, tornadoes, fires, floods. And so, are you protecting your data? Are you protecting your product in the event of an event, something like that, a natural disaster happening? And it seems like a fringe case, but it happens all the time. There were all of the fires in LA, and of course, there are a lot of data centers in LA. There are hurricanes in Florida, hurricanes even reaching up into Atlanta. Um, I know that we even use data centers in Atlanta, Georgia. And so there are problems that can happen out there. I am sure about half the world's data at this point is in San Francisco and someday the ocean is going to, you know, big enough earthquakes going to happen and San Francisco is going to get swallowed up by the ocean. And so what is happening to all the data then? Well, that goes to the planning portion and that TPLC. I think if you need to have the data always be available, uh, then you need to find two data centers that are the data is mirrored on and the data centers are geographically pretty far apart. Yep. I think that the in the DoD, I think it had to be like 150 miles apart. I think in general like a natural disaster is contained to like 150 mile radius was a the rationale. That sounds about right. I would probably put them more than 150 miles apart, though. Yeah, I figure you would want to be in two pretty distant places. Like if you have a data center in the southeast, then you would want one in the northwest or stick it in the middle of the desert of Nevada or something where nothing really happens, right? Yeah. And so that that is part of the Total Product Life Cycle. What are what are some other common things that people neglect to think about from a TPLC perspective? We talked about the development environment. We talked about updating. We talked a little bit about decommissioning. We talked about maybe where you choose to host your data or your software. Um, what other considerations are there from a TPLC perspective? So, I think thinking about how users are interacting with the product and where they are interacting with the product. Um, this will tie in a little bit to the update side of things, but what if users are not always connected to the internet? What if they are in an environment where they cannot receive updates? What if their infrastructure is not going to support your device in the way that it may be supported in other places? A problem that we recently were dealing with for one client was they had a solution. Their device had a problem going back and forth with certificate problems. And what the FDA wanted to see is that certificates were essentially baked into the device and a certificate authority in the hospital network could upload that certificate in it would be part of the certificate authority. There is a trusted route going back to that uh CA. And what the problem was with the manufacturer is not all of our clients, not all the hospitals that we are operating in have a mature enough IT department where they have this full infrastructure set up. So now what? we cannot cater to, we cannot change the device for only a few certain environments. We have to find out a good solution that is going to work for everyone. And so we ended up finding a good solution which was essentially if a hospital does not have a CA that they can use, the um manufacturer was going to provide basically a script to develop that CA themselves and then pull the certificate onto the device. So, they acted partially as the IT department, but then there is a whole additional layer of involvement. And so now you have a script you have got to make sure it is secure as well. Exactly. Yeah. Without a hard-coded password that ripped in there. And so a lot went into this back and forth. It was a pretty difficult situation to cover. So, thinking about where your device is going to be in its entirety is an important part of the Total Product Life Cycle as well. And if we say the SPDF is a subset of the TPLC, which I agree with, you mentioned uh the hard-coded password uh that was someone just grabbed off the internet like and this happens a lot. People f on Stack Overflow or whatever forum that people are writing about code, people just software developers grab stuff straight off the internet and put it in the code. What are some of the best ways to set up that Secure Product Development Framework so those things can be catched sooner than later? So, what we want to look at is first off what practices are your engineers following? You should have very very strict standard operating procedures. This is how you are writing the code. This is the you know standard that we are adhering to. If you are using, you know, let us say you are writing a Java application, there are secure coding standards like the Cert Java standard. If so, you want to make sure that engineers are following a principle such as that to the letter. There should be checks in place. Um, no code can be changed without multiple layers of verification from other people. You always want a couple sets of eyes on something. And so, like even if, you know, when we are writing documents, we always want someone to double check. oftentimes I am going to be the one who is writing off signing off on someone else's pen test report. And so even though I am the reviewer for that report, if I am writing my own pen test report, I am going to have someone else go and review it because we always want multiple sets of eyes on something. Um, so adhering to secure coding standards, multiple steps of verification and checks and then a CI/CD setup. So, that is continuous integration and continuous development. As you are making changes to the code, it is going to continuously push out these changes to one place or another. And as you are going in, one of those places can be static testing where it is scanning through the source code, looking for hard-coded credentials, looking for insecure coding practices, and then it is flagging them and saying you need to fix these. Uh, you also want to check the supply chain security. So, going into that Software Bill of Materials, how are you not introducing any insecure components into the device? How are you checking for these insecure components? So, that can be an automated process as you are going through the code flow. You need the tooling and the infrastructure in place. But then as soon as you are making changes, everything is automatically checked and verified. So, you do not really have to worry about missing something like that. Even if someone is making a mistake, they are not following these secure coding standards. You will have automated tooling to make these checks and you should have multiple sets of eyes that are making these checks as well. So, in a good environment, mistakes will be caught and remediated very quickly. We are not saying that no one is going to make mistakes. That is just not feasible. But you should be able to find them. How how likely is it that people can actually set up these Secure Product Development Frameworks or a secure CI/CD pipeline? You know, how feasible is that actually? Because it seems like most development organizations that these could be even be a CRO or internal development do not have any of this. No. And it is true. It is it is not easy when you are writing code. If you are just going in writing code, pushing it out to GitHub, it is not going to be a secure process by default. What I can say for sure is 100% of engineers will make mistakes. Everyone is going to make mistakes when they are writing code. It is impossible to write perfectly secure code as soon as you are moving past one or two lines of code really. And so setting up that CI/CD pipeline, that Secure Product Development Framework, it is not easy. It is going to take time. It is going to take effort and it is going to take man-hours. You are usually going to see people do this just explicitly as a job. They are architecture engineers or they are, you know, CI/CD engineers. All they do is develop that development framework, which is kind of interesting. They are developing the developers, but making sure that the process is built out because the developers are not going to want to have to set that up themselves. They are going to want to write the code. This is a special set of skills that you need a specialized individual to do. Yeah, that is a good point. I think from I would say if I had to add up a number on it, uh 10% of our clients actually have something that is pretty close to a Secure Product Development Framework before they come to us and we can help them. Five. You think it is five? Oh, yeah. I cannot. Well, you deal with our clients more than I do. Yeah. even down to static testing as a part of continuous integration. It is a standard best practice with development. But the issue is we are interacting with a lot of startup startups who may not have the expertise or the you know background in this to understand how to develop all of these cybersecurity concepts in the pipeline. And so when we are looking at something like static testing, which is essentially going to be secure coding 101, they may just be too new in the space. They may not have the resources. They may not have the manpower to do this. And so it is pretty uncommon that we will see a good, it is pretty uncommon that we will see any sort of Secure Product Development Framework. And it is extremely uncommon that we will see a very strong one. Now having said that, usually when we are working with bigger company, strategics in the space, they have the maturity and they have the resources to have engineers focused purely on the development architecture. And so someone is maintaining all of this infrastructure. They have it stood up. Every single piece of code goes through all these different layers of checks and they have full-time employees just making sure that infrastructure is in place. And so it is not really that people are manufacturers are being lazy or they are forgetting about it. It is just that it is very hard to do and very time consuming and there are a million things happening with a startup. So, sometimes it can get pushed to the back burner. It is interesting because uh I know a lot of startups. We work a lot of startups and a lot of them have contracted out their software development to a software development organization. And then when we peel back the uh covers a little bit, that software development organization, which is all they do, they do not do secure software development, which I always find interesting. Uh unless it is like a highly regulated industry like nuclear. Uh it is still like in its infancy with with MedTech and in I think software development in general. Yeah. No, I definitely agree. And it is it is really interesting because I think that creating software documentation that has been something that has been happening since the 90s. And so contract manufacturers, contract engineers are pretty used to creating good software artifacts at this point. That is why when we are looking at if you look at a uh contract manufacturer for MedTech, they will often say we are ISO 13485 compliant, which is quality systems and medical devices. What that means is they are preparing all the documentation and artifacts to feed into your quality system saying you know here is our architecture here are requirement specs here is our VNV and traceability back to the requirement specs. Now what we want to look for is are you IEC 62304 compliant? And 62304 is medical device secure or secure medical software development. And so that is when you are moving into that CI/CD pipeline. They have all of these checks in place. They have all the static testing the analysis of your Software Bill of Materials. They are building in all of the security requirements and they are testing them, which is required to show the FDA security requirement testing. And so, but interestingly enough, if you look around for contract manufacturers in the MedTech space, you are not going to see 62304 compliance as much as you think you would, but you will always see secure soft or not secure, but good software documentation, good cybersecurity documentation is a little bit newer. Yeah. I wonder if this uh challenge is ever going to or this gap is ever going to close because I I have been in involved with cybersecurity for like 30 years and it is always been a challenge getting software developers to develop code securely even though there is plenty of documentation today like OWASP has a whole DevSecOps playbook, you know, it is not like there is a lack of what to do. I think like you said it is it is costly like a lot of these SaaS tools are extremely costly and um it requires someone that is looking at a bigger picture than maybe the person developing the code because they are looking at this overall framework and how it should all tie together. Um, yeah, so I think it is uh just the complexity of implementing maybe the cost are a couple of the reasons it is not being done as much as it should be should be. Definitely. Yeah. And we always say this, but cybersecurity is no one's first choice to do. Nobody wants to do cybersecurity. It is expensive and it does not add any tangible immediate value to your product. Now, if your product gets hacked, you are immediately going to see the value of cybersecurity, but nobody wants it until it is too late. And so when someone is trying to roadmap the 10,000 things they have to do when they are trying to start up and create this product, MedTech Innovators, a Secure Software Development Life Cycle, it is expensive. It is very time consuming and it does not add tangible value. They are going to look at things like marketing, sales teams, having an actual product as the immediate priorities. And so very often this can just slide to the back. But it is a little bit of a difficult situation because if you do not have it in place, you are not going to meet regulatory scrutiny. You are going to have problems getting acquired by hospitals or resellers. And so, yes, it does not provide immediate visible value. The second you need to meet regulatory compliance, you will see the value. Yeah, that is a that is an interesting point because I think the number like I have heard from uh investors is like 93% of MedTech startups fail. And imagine like you are trying to spend all this money on setting up a Secured Product Develop Framework, a TPLC and decommissioning, but your product actually never gets launched. You know, that is kind of irony is you need all these things set up to get approved by the FDA or EUMDR, but then if you have not actually figured out how to commercialize your product, these things that may have all been for nothing. I guess just like the development of the product or launching the company, it is the same concept. But yeah, it is it is interesting like if your product never gets out there, the decommissioning aspect is irrelevant, right? But you are not going to get your product out there if you have not considered the decommissioning aspect, right? Yeah. So, I I can understand like why the why these MedTech startups are like we do not even know if our products are going to make it, but we are we are having to spend money to figure out how to decommission it if it makes it, but we are just trying to get it to market. You know, it is it is it is a tough it is a tough decision and but you have to like do it for the regulatory requirements. But from a a funding and a getting something to market, I can understand the the mindset around that around like cybersecurity in general, like why they why people do not want to do it and why there might be some resistance about even these things like the SPDF or the total product uh life cycle. Totally. Yep. Cool. All right. Well, I think we are coming up on time here. Any last words of wisdom, Trevor? Yeah, I think that uh cybersecurity should start at the concept and then it should finish at your exit. So it is not something you do once. It is something you are always considering and that is the Total Product Life Cycle. You have to figure out how you are handling cybersecurity all the way up until, you know, the very end. Even something that we are preparing documentation for as part of our postmarket plan is what happens if the company goes under to any fielded devices. And so, it is a hard conversation, but it is something you have to think about. And so cybersecurity needs to be addressed throughout everything. That is why they call it the Total Product Life Cycle instead of partial product life cycle. You know, that is a good point. If even if your company uh you you could have filled like one product and your company goes out of business, what happens to those products, right? Yeah. that is one of the use cases we have to walk through. Yep. every single time we have to go through that conversation and uh clients never want to have it. They are like, “We do not want to think about what happens if we go out of business.” And I say, “Well, you have to. You have to think about all this.” It is interesting because people do not want to think about their own death either, right? Because I know I was talking to some people like not too long ago and there this woman's father died, did not have a will, did not have a trust, nothing was figured out. So, it was a mess. the family uh did not know what to do with anything and it caused more stress, right? And then about a year is gone by with this this person and I have asked her if she has got a will and she does not. So I am like, you just went through this with your parent. It was very stressful, but now you are going to put your kids through it. So like, you know, it is like I am I am saying this is like the Total Product Life Cycle decommission ourselves, I guess. But it is like we often do not think about it. It is like but but it makes it it makes it more challenging for everybody else. There is resistance though. Oh, yeah. Yeah. Nobody nobody wants to have that conversation. Nobody wants to talk about their company failing. Nobody wants to talk about themselves failing. But but statistically we are all going to fail at some point. So we might as well figure it out. At least for me I am doing uh, you know revocable living trust and a few other things. I am getting those things done because I do not want to have leave all this confusion to people if I die. Uh, and I have witnessed other people that have had that happen to them. So, I am just taking care of business. I guess I am thinking of the TPLC in terms of me. Yeah. With all the uh dangerous sports and extreme whatever that I do, I should probably figure that out sooner or later. Yeah, you should probably figure it out, too. Yeah. What happens if I go down free diving but do not come up? Who Who gets the car? All that good stuff. Exactly. Otherwise, it is up for up for uh decisions and people like do not actually know what your intentions were at some point. Yeah, exactly. Guess. Cool. Um, all right. Well, I think we will wrap up here. Thanks everyone for tuning in to the Med Device Cyber Podcast. I hope you found value in this podcast about the TPLC and a little bit of a deep dive into SPDF as well. And hope to see you on the next one.