Skip to content
InTechnology Podcast

#9 – The 411 on PSIRTs and the Incident Response Framework

In this episode of Cyber Security Inside, hosts Tom Garrison and Camille Morhardt talk PSIRTs with Director for Red Hat Product Security, Pete Allor.

 

Pete has an impressive history with PSIRT and the convo covers things like:

•  What a PSIRT is and why you need to have one

•  The dangers of falling into a technical trap when addressing product vulnerabilities or problems

•  Evaluating risk and scoring vulnerability

•  Why incident response requires organization-wide coordination and communication

•  The Incident Response Services framework

•  The role transparency plays

•  And more

 

Plus, Pete’s got a book recommendation, Camille’s got a tip for musicians forced to play in the dark, and Tom’s got a trivia question that will get you free drinks, every time. Check it out!

 

Here are some key take-aways:

•  PSIRT is designed to address vulnerabilities with the company’s own products. If you don’t have a Product Security Incident Response Team, you need one.

•  Get to know your auditor well and figure out what your risk tolerance is internally.

•  PSIRT should be proactive, not reactive.

•  We need to talk vulnerability scoring, so we have a commonality, a base to work from. And we need to talk about severity, as in risk.

•  PSIRT is a coordinated effort. You need to give everyone the information they need, which means removing the technical babble and simplifying.

 

Some interesting quotes from today’s episode:

•  “So fixing everything is not really the smartest thing because it’s not necessarily a problem.

•  “Now we’re looking at ‘how do we respond?’ It’s not just disclosing to us. It’s a matter of publishing to our customers. But what we are publishing isn’t a vulnerability. We’re publishing a response. And that’s the change in discussion we need to have.”

•  “A customer can’t do anything with the disclosure. They do everything with the response. And it’s in our name: Incident Response. So we’re giving them the tools to mitigate, we’re giving the actions to mitigate, and we’re giving them the code.”

•  “I think we all fall into a technical trap sometimes, which is we focus on the issue itself. You know, this is broken. It’s a TLS issue. It’s a DNS mask. It’s, it’s something like that. And so we look at it from a very technical aspect. What we lose sight of is that it’s an engineering response.”

Share on social:

Facebook
Twitter
LinkedIn
Reddit
Email

Announcer: Welcome to the cybersecurity inside podcast. Cybersecurity is not just for coders and hackers, CIS owes and their executive peers need to think about cyber security differently in the cybersecurity inside podcast, we discuss relevant topics in clear, easy to understand language. Now, here is Tom Garrison.

Tom: Hello, and welcome to the cybersecurity inside podcast. I’d like to introduce my co-host Camille Morhardt. Hi, Camille, how are you today? 

Camille: I’m great, Tom, how are you doing?

Tom: I am doing well. Are you still holding out at the Oregon coast? 

 

Camille: I’m not at the coast today. I’ve come in, um, protected and masked into the city to tread amongst the English briefly.

 

Tom: (laughs) Very, very good. So what, uh, what, what sort of interesting topics do you have in mind for today.

 

Camille: As you might think is natural coming into the city and being afraid. I’m thinking about incident response. So there’s a group of people who reside in some companies who call themselves product security incident response teams, which is PSIRT for short.

And these teams are designed to address vulnerabilities that are discovered in the company’s own products. So as I think a little bit about this, my question is does having these teams indicate an institutional expectation of failure? 

 

Tom: Hmm. That is, uh, as normal. It’s pretty deep topic you’ve got there. Um, you know, I, I think about it this way. You have to plan for the unknown. And you know, one of the things that I like to think about is an analogy for people that aren’t in the industry, when we’re building our chips, Intel’s chips, it takes us about four years or so to architect the chip. And then it takes us once we’re done with the design, it takes us give or take about a year to manufacture the chip.

And then we find launch it and it goes out into the wild and those devices, whether it be PCs or servers they live for five-ish kind of years out in the wild and over that time science advances. So whether that be the latest in security attacks or new use cases for the platform to be used in ways that we never envisioned, even when we designed it five years ago.  At the end of a device is useful life in the wild, that device was started to get architected 10 years ago. So a lot of science advances between them. And so you need a way to be able to update the platform. Along the way based on what whatever is, uh, you know, a modern use case or a modern class attack, you need to keep that device safe and updated.

 

Camille: And I suppose even longer when you’re talking about embedded or internet of things, products, we’re thinking about vehicles or some kinds of factory systems. 

 

Tom: Yeah. Industrial applications, those, those tend to have much longer lives as well. So yeah, you can be up in the order of like say 10 plus years of useful life for those. Whereas PCs, for example, tend to be, you know, usually around five years. 

 

Camille: So when I, when I think about, you know, getting a software update or an OS update, I’m usually kind of excited about it because I feel like, Ooh, there’s going to be some kind of a new feature coming. And what I really want don’t think about is the fact that they may be mitigating vulnerabilities along with that update–in fact, that may even be why they’re releasing that update at that time. Um, and you know, when it comes to hardware, First of all, I’m not really used to even thinking about updates in the hardware space. And second of all, if there is an update, if anything, I’m potentially going to have a performance reduction as opposed to improvement in general, that’s kinda how I think of it.

So, you know, how, how should we be thinking about hardware updates in this context? 

 

Tom: Yeah, I think about the hardware in, in a couple of different ways. But the reality is kind of. That sort of analogy I used before in terms of how long it takes, you know, you want to keep your device safe, so you need to update it.

And that is a, that’s a behavior change that really we’re still in the middle of in the industry. So some people, when they buy their PC, whether they be consumers or whether they be IT folks, they would kind of just prefer, I guess if they, if the device just, they bought it, they loaded whatever software on it and then they just used it over the life. When in reality, that’s just not the case. We need to change the behavior that people are used to to keep being those platforms updated. Uh, that’s number one. 

Uh, number two, there are cases where some of those updates have an impact to performance, but usually it’s very small. It’s not usually a significant impact. And sometimes we do actually add features for these updates. And so we will have embedded the capability. In the hardware, but we’ll have it dormant. And so at some point, maybe there’s a new operating system version that comes available or applications that are now available that take advantage of a feature.

And so we’ll use this updating mechanism to actually enable a feature. So all of those use cases are our reasons why we do updates for hardware. 

 

Camille: I have so many more questions about this, but I think I would, I would really like to hear from somebody who sits in these products, security incident response teams, about what kinds of things come in and now how important transparency is, or isn’t as you’re dealing with this type of thing and how we kind of keep everybody secure if that’s the goal.

 

Tom: Let’s do that today. Today’s podcast. Would you say? 

Camille: That sounds good. 

Tom: Okay.

 

Our guest today is Pete Allor. He is the Director for Red Hat Product Security and has an impressive background, including board of directors and councils such as the IT Sector Coordinating Council. He worked on the National Infrastructure Advisory Council focused on vulnerability disclosure guidelines and is currently the PSIRT Special Interest Group chairman. And also he’s a retired Green Beret. So welcome to the podcast, Pete. 

 

Pete: Thank you, Tom. Appreciate having me aboard.

 

Tom: Maybe it would be best to start today with just a little bit about, you know, I mentioned the PSIRT, maybe we can start with what PSIRT is and what you do with PSIRT?

 

Pete: PSIRT is really focused on how do you respond to product problem sets.  So when a finder/researcher discovers an issue, they normally bring it straight to the vendor and the vendor operates a PSIRT to respond to that disclosure and to work it through with engineering to remediation. 

 

Tom: Okay. And, and so, um, when we, when we say PSIRT just for everybody, if you don’t remember, it’s a Product Security Incident Response Team.  Now, is there consistency within the industry on how you do PSIRTS or how does that work out?

 

Pete: No, I think we stumbled over time to a certain degree of, uh, those who really understood it. It was just a few large vendors and those were like, how do I tackle this? And that’s one of the reasons that first went ahead and, uh, commissioned the Special Interest group to work on PSIRT. We took several years to develop a or services framework.

So it’s just a framework of how to do instant response, what to think about not, not how to do it per se, like instructions, but things you need to consider. And the interesting part was as we developed that, and we took it back to the board directors for the review and approval and publication, ne of the things that we learned was that the vast majority–about 70% of the board had no idea what a PSIRT did. Nor did they understand the amount of importance that went to their PSIRT operations. So we’ve discovered in the first order, people didn’t understand it. 

 

Tom: Interesting. So these people that were the board level, folks didn’t even understand what the vulnerabilities were or the exposure to their business. Is that, is that the right way? 

 

Pete: I think they understood a little bit about the exposure. What they didn’t understand is how do you coordinate that? How do you, how do you go about the business of taking that in and massaging through all that, finding a fix and publishing that? And it’s almost as if that was a black box.

 

Tom: And when was that Pete? Roughly? Like what year are we talking about?

 

Pete: Oh, we started that in 2014, as a concept. We started in earnest and right in 2015, and we didn’t publish almost until 2018, our Incident Response Services Framework. 

 

Tom: Wow. That’s shockingly recently. 

 

Pete: Yes, it is. When you consider that, that should have been around since like, uh, the Morris worm.  So should have been around from the inception. 

 

Tom: Yeah. Interesting. So, um, do you think now, uh, in terms of, you know, the engagement with the broader industry, do you think there is more of a maturity in terms of how to take from issue awareness through the process to mitigating whatever the issue is? Do you think there’s more understanding now across the industry on how to do this in a coordinated way?

 

Pete: I think they will understand more about the need. I think we all fall into a technical trap sometimes, which is we focus on the issue itself. You know, this is broken.  It’s a TLS issue. It’s a DNS mask. It’s, it’s something like that. And so we look at it from a very technical aspect. What we lose sight of is that it’s engineering response. So if you look at your Security Development Life Cycle or SDLC, this is the last part of that entire cycle. It’s what happens when everything sort of, kind of goes wrong, so you go to fix it. So it’s very reactive. What we had to learn though, is that you’re proactive in your planning and more importantly, you’re proactive and understanding who you need to work with your stakeholders internally.

 

Camille: Is this more about assessing risk or is it more about addressing vulnerabilities as they come in? Or are those the same things? 

 

Pete: There’s a real unique problem set. So if you look back in the early 2000s, we started talking about all the different scoring systems everyone had. And some people scored on a 10 point scale, some scored on 180 point scale, but they started at 20. That was through CC for instance. So we had a mismatch. And so from the NIAC, as we look at vulnerability disclosure, we came up with, we need a common vulnerability scoring system. Well, that took a few years to develop it and we came through and we’re now in version three, looking to go to version four as an industry.

So now we know how to address those vulnerabilities. And I think that was good, but we got lost.  Because in reality, what a PSIRT does that helps address risk and severity. And you have to look at severity is to the product and you had to look at severity as is to the customer. And now that conversation is starting to morph some more to address risk. Cause that’s really the, when you talk it externally, you need to talk risk. And so you’re seeing this starting to publish in some of the more advanced, thoughtful guys who are leading the charge of, we need to talk vulnerability scoring, so we have a commonality, a base to work from and we need to talk about severity as in risk.

And so we’re, we’re providing both sets of scores. 

 

Camille: And this is because you couldn’t possibly fix everything? or because people wouldn’t possibly adopt all the fixes? I mean, why not fix anything that comes in as an issue? 

 

Pete: It sounds great, but I think in, from an instant responder, I’d love it from a customer perspective, absolutely want it.  But in a reality and what it costs, you couldn’t approach to do it and using some rough figures that we’ve looked at internally, probably a quarter of what we look at our “criticals” in importance on the severity scale. Probably a little over 40% are really on “moderates” and the rest are “lows.” So if you really look at it from that perspective, you are going to suspend double the engineering and get diminished returns, especially as it regards to risk. 

So fixing everything is not really the smartest thing because it’s not necessarily a problem. 

 

Tom: Yeah, I think so. Let’s just pause for just a second and think about, uh, not think about, but maybe describe for the listeners, some of the terms that we’ve heard.  We’ve heard the CVSS so the, the scoring that we have, and, and so for the users, maybe we should describe as a 10 point system. And we look at various aspects of the severity of the issue or what data is at risk or other elements. And obviously the higher, the risk, and the more significant the issue is the closer to 10 you get.

 

Pete: Yes, and but. So listen, talking about CVSS, the Common Vulnerability  Scoring System. Back when we did version one, we had a hard time about matching the 10 point to 180 point scales. We ended up breaking it down to three areas. There’s a base score and all these are zero to 10 or one to 10. And the base score is what we call immutable. It doesn’t change. It’s there. 

The next one we talked about was temporal score. In other words, there are factors that happen that would change that–an exploit code being released, a patch being released, how you look at severity and how the component is put inside a vendor’s product or an open source project that changes it. So there’s a temporal score. And the first one is well understood and published by the vendor. The second one has a lot of variables, which change over time and it’s real hard to keep up and maintain. 

The final section was called an environmental and environmental score is all about how it’s deployed. So if the customer deploys it and they put it behind the firewall and IPS, and it’s really not containing a lot of sensitive data, there’s another score. Hardly anyone does that last part. 

So in CVSS, we’re trying to make sure we all do the base. And we try now to address the temporal score–and believe it or not, that’s been not there almost what, 15 plus years, 17 years and we’re just getting around to really buckling down and doing the temple score. And so that’s why the severity rating from everyone is so important because we all went to the severity rating as a means to shortcut the temporal scoring, which didn’t keep pace with technology. 

 

Camille: When you start getting into some of these details, which are fascinating, by the way, I assume one of the things you would care about with respect to vulnerable vulnerability disclosures would be transparency.

 

Tom: Yes. 

 

Camille: So how do you balance, you know, communicating with people and this level of complexity? 

 

Pete: One is there’s an over arching desire to be simplified. Due to the technical issue, there’s so much there that has informational that you need to do what we’re trying to do with all these different nuances. We are trying to coordinate through the entirety of a corporate entity or an open source community and that, and navigate through all those and work with our stakeholders to address those.

So you’re talking to corporate communications, you’re talking to maybe analyst relations. You’re talking to legal, you’re talking to engineering. You know, you have to coordinate not only with the development part of engineering, but the quality assurance, quality engineering of what was there and the delivery mechanism. And of course do not ever forget your TAMS and your, your customer support, because that is the face that the customer going to call. 

And so you’re trying to put it as much information there and coordinate it. So what you really come down to is do you do a PSIRT full-time or part-time. And with the nuance and the more products and services you have, the more your demand for a full-time PSIRT really comes to the floor.

 

Tom: There’s probably two sort of, at least major takeaways that I can think of and there’s probably more, but one would be just for any of the listeners that are working at a company, whether it’s a hardware or software company, the notion of having this capability of a Product Security Incident Response Team is something that you need to have. If you don’t have it, you need to have it. 

And then the second piece is really I’m thinking to the customer, the end customer. There has to be an understanding or an appreciation at least without necessarily having to be an expert of what is the output of these PSIRT teams and how, what do I do with that information now that it exists? I don’t want to throw my hands up in there and say, ”this is too confusing. That’s for the experts to deal with.” 

I’m wondering if you can give some advice on the non-experts that are listening. Like what do I do with this information? 

 

Pete: So for the end user customer set, um, they’re going to have a lot of discussions with their auditors. And I think I would say, get to know your auditor as well, because you have to figure out what your risk tolerance is internally. What you’re really looking for is the vendor bowls and it comes out and breaks down all the information in there. For instance, in my organization, we are now writing to an executive as part of our summary. We try to remove the technical babble out of there. 

The other part you have to look at is how do I understand all this? And, and one of the things as a group of PSIRTs–and the nice part about peace is we’ve learned to work together. We may be competitive in the marketplace, but because components are so broad across everything, for instance, there’s the microcode issue, you know, multiple, multiple, uh, CRO folks and get together and then we’re going to work with people downstream to work that prior to release. 

So when a customer is looking at it, he’s saying, how do I consume it? What we came up with a, we call it CSAP is a Common Security Advisory Format so that you can consume all that and a json or even a CSV, if you want, and say, I can read everything I get from every vendor the same way. And so your people know how to prioritize, what needs to happen. 

 

Camille: Do you consider part of PSIRT to be not just managing the vulnerability, but managing the disclosure process?

 

Pete: We manage– and I think this is something that we’re realizing, uh, across PSIRTS, where we need to change the discussion just a tad bit. Disclosure really has the initial part of what you bring to us. And once we started our triage process and we’re looking at it and we’re assessing the severity of this and how fast a response we needed coordinate that, if it is more than just our product, We have an ability to reach out to others and say, “we have a group problem set.” And generally it’s three or more vendors, we’ve started to reach out to others that way. 

So now we’re looking at how do we respond? It’s not just disclosing to us. It’s a matter of publishing to our customers, but what we are publishing, isn’t a vulnerability. We’re publishing a response.  And that’s the change in discussion we need to have.  A customer can’t do anything with the disclosure. They do everything with the response. And it’s in our, it’s in our name. Incident Response. So we’re giving them the tools to mitigate, we’re given the actions to mitigate, and we’re giving them the code–

 

Camille: You’re talking about like patches or code patches or things like that, when you say response? 

 

Pete: That’s the latter part, but yes, exactly.  Because the customer wants to fix, they want to know, is it easier or harder to fix? Do I need to do a restart everything or can I just do it while a hot fix it? You know, while everything’s running. And do I need to do for my next major update cycle? Or do I need to do it right now? 

That’s what they’re looking for.  They want that level. Tell me what is, I really need to know, take the guesswork out for me. I have to still decide, but take the guesswork out for me. 

Camille: Are there different levels of transparency in this field? Like some companies or kinds of companies are going to tell you about all the vulnerabilities or all the risks that they’re aware of, even if they discover them internally? or they’re discovered externally, but they’ve decided not to fix them because it’s not a big deal? and other companies or other kinds of companies or industries only tell you about it if they’ve got to fix or only tell you about it, if it reaches a certain level of criticality or discovered externally? Are there breakdowns like that or is everybody on the same page? 

 

Pete: There are definite maturity levels by industry and by companies, people are pretty new. We are in a tech area are I think pretty avant-garde about components and understand that and being able to respond some longer-term organizations are a little stuck, but overall in the IT industry, we’re doing pretty much leaning forward. 

You get to some other industries and it’s not until, you know, something breaks and everyone knows about it or in some industries, it’s only when it’s very critical and we really, they have to.  Um, but you know, what we’re doing now is we’re talking across the industry. And because of industrial IOT and the OT blend into the IT world and where we are in the IT world, you’re starting to see a blending and some who were before not telling at all are starting to talk and they’re starting to move into an incidence response approach.

 

Tom: So Pete, we have. Fun little segment that we do as part of the podcast and it’s really just sharing a cool new piece of insight or a new, recent learning that you’ve had. So do you have anything along those lines? Something you’ve learned recently? 

 

Pete: Um, yeah, I’m trying to pick between two. So, um, I’ll go with one where we actually did a group read of a book at work, and it was kind of interesting.  The name of the book is Turning the Ship Around. I don’t recall the author right off the top of my head, but that was a Navy guy and I’m an army guy. So it was like, “Oh man, I’m going to have to read a Navy guy book!” (Tom laughs)  Oh, and he was a submarine commander. 

But as I read that book, I went, “yeah, yeah. Oh yeah. Oh man!  I did this!  This, this guy put it together!” And the interesting part was I saw people who weren’t not military, but they were going like, “do you guys really do that?” So if you’re looking for how to work with a team and think differently in an organization, go read, turn that ship around or Turn the Ship Around. To me it’s just a unique way of looking at how to have people inside the organization, contribute to the solution and not have to be hierarchical down driven. 

 

Tom: That sounds great. I got to look that one up. Camille, how about you? 

 

Camille: So, uh, on the line of, you know, useful tips that you can do. Um, when you’re in a, uh, I was watching an interview with Ida Nielsen who was Prince’s, uh, bass guitarist for about, I think, six years.

And the problem that she had was he, they would go completely dark in between. Songs during a show so black that she couldn’t find herself on the fretboard. And then he would start with whatever he started with. So she never knew, you know, there was not a set list, they drafted one, but he never stuck to it.

Uh, and so she needed to know exactly where her hand was. So she’d hit the right note perfectly the first time, and this is Prince it’s gotta be right. And so she had a, uh, a bass custom designed for her where. You know, those little dots on the edge of the fret board on the side, that kind of tell you where you are.

If you’re on the fifth Fred or the 12th fret, she had those put in glow in the dark 

Tom: Oh, 

Pete: Range Rise. Yeah. That was glow tape we’d have sown into the back of our caps and they were called range rise. So you could see the guy in front of you. 

 

Tom: There you go. So I am going to go, I I’ve been for the last several podcasts, sort of sticking in the world of entertainment and, and I could do that, but I don’t want to be typecast as the entertainment guy.

I’ll probably go back to it because I love entertainment stuff. This time I’m going to go into the world of trivia. And it’s a trivia question that I just think is fantastic. The question is, can you name the largest us city by population East of Reno, Nevada and West of Chicago, Illinois?

 

Camille: Houston, 

 

Tom: Houston. Okay. Pete, you have a guess?

 

Pete: I wouldn’t have thought of, uh, Houston, um, I don’t know. I’d just be different. I’ll go Oklahoma City. 

 

Tom: All right.  Well the correct answer and get ready to have your mind blown. The correct answer is Los Angeles.

 

Camille: WHAT?!?

 

Pete: What was the first part of the question?

 

Tom: “It’s East of Reno, Nevada. And West of Chicago.” So East of Reno, Nevada. Yeah. If you look at the map, uh, California has that kind of 

 

Camille: Pause while I Google. 

 

Tom: But anyway, it’s a, it’s a fun little trivia and you can make real money and it’s a great bar drinking game. Cause you get free drinks all the time with that question. 

So, uh, just to wrap this up, so Pete, I do want to thank you very much for joining us today was great to learn about, you know, product incident responses and how, um, you know, people can take lessons from your experience as well as, uh, you know, how to interpret issues as well.

So it was great to have you on the show and thanks for joining us. 

 

Pete: They very much, I really enjoyed it. Uh, I look forward to listening to a lot of your other podcasts. Uh, as I know about this, this is just a wonderful thing to get tuned in. 

Tom: That’s great. Well, tell your friends. All right. Thanks for joining us, Pete.

And we’ll see everybody else on the podcast. In the future,

Announcer: subscribe and stay tuned for the next episode of cyber security inside. Follow at Tom M Garrison on Twitter to continue the conversation. Thank you for listening.

 

More From