Search
Close this search box.
EPISODE 146
Security Metrics

EP 146: Security Metrics

Our bi-weekly Inflection Point bulletin will help you keep up with the fast-paced evolution of cyber risk management.

Sign Up Now!

About this episode

December 5, 2023

How can we measure success with cybersecurity? Let’s find out with our guest Jared Pfost. Your hosts are Kip Boyle, CISO with Cyber Risk Opportunities, and Jake Bernstein, Partner with K&L Gates.

See Jared’s “Security Metrics Reference” here — https://www.cr-map.com/metrics

Tags:

Episode Transcript

Speaker 1: Welcome to the Cyber Risk Management Podcast. Our mission is to help executives thrive as cyber risk managers. Your hosts are Kip Boyle, Virtual Chief Information Security Officer at Cyber Risk Opportunities, and Jake Bernstein, partner at the law firm of K&L Gates. Visit them at cr-map.com and klgates.com.

Jake Bernstein: So Kip, what are we going to talk about today in episode 146 of the Cyber Risk Management Podcast?

Kip Boyle: Jake, we're going to talk about something that we probably should have talked about, made a whole episode about a long time ago, but that's okay because we're going to fix that right now. And we're going to talk about security metrics, which is another way, a fancy way, of saying how are we going to measure our success with our cybersecurity programs, our information security programs? And, I could think of nobody better to help us navigate these treacherous waters than our guest Jared Pfost. And so, Jared, we're so glad that you're here to help us sort this out. Sorry it's taken us so long. We've been wanting to have you on the show for quite some time now. But you're here, so thank you.

Jared Pfost: It's a pleasure to be here. Hello, everyone. I'm Jared. For a quick introduction, I am passionate about building information security teams and products. And nowadays, I just say, over 20 years of learning. I have experience across all the information security domains with expertise in building secure development services, GRC measurements, and visual storytelling. Currently, I'm consulting while looking for my next full-time opportunity.

Jake Bernstein: Excellent. And, Kip, you and Jared know each other. Maybe just let us know what your history is.

Kip Boyle: Oh yeah, happy to do it. And, it's a good story, I think. So, Jared and I bumped into each other quite some time ago. I would have to mumble the actual number of years in order to stay consistent with Jared's philosophy of stating how long we've been doing this. But I was the chief Information security officer at PEMCO Insurance.

Jake Bernstein: Okay. So now I know about how long ago it was.

Kip Boyle: It's been a while. And, Jared was building this fantastic software product. And it was meant to help people such as myself in that position to do a good job of identifying our cyber risks, and then to manage them. And, he was a solopreneur, is the way I saw him. He was building this cool tool. And so, I signed on as a beta customer, and had a really good experience doing that, and really enjoyed the times that Jared would come by to the PEMCO Insurance building in Downtown Seattle, and we'd talk. I'd give him feedback on his tool, and we would talk about where he is going to take it to next. And, it was really fun, because not only was it helping me do the job that I had at the time, but it was also fun because I really just appreciated how clear-minded Jared was about what he was doing.

And, I was still a closet entrepreneur. So, I thought it was really cool to be talking to somebody who was out in the world talking to customers and trying to get that product market fit sorted out. Anyway, that could go on for a long time. It was fun.

Jake Bernstein: And, was the product about security metrics, or was that something that came later?

Kip Boyle: I'll let Jared talk about that.

Jake Bernstein: All right.

Jared Pfost: That did come later, but it was an integral parts, and what we'll talk about more of what is the role of measuring our current control posture and how effective should we be?

Jake Bernstein: So, I'm going to start with a very basic question. And Kip, you're right, we have been remiss in talking about security metrics. So, Jared, what are security metrics and why should people care? And, I will start with a hypothetical or an unfair question, which is, isn't it impossible to measure a negative like a data breach that doesn't happen? Or, is that not metrics?

Jared Pfost: Let's start from what the textbook says, we're managing risk to an acceptable level while minimizing cost. What is an acceptable level of risk? We know the types of impacts that we're working to avoid, measuring our current controls helps inform the likelihood of those impacts occurring. The fun part is working with business leaders to determine what is an acceptable level of control performance. And, those two pieces, where are we today? Where should we be? That's where metrics come into play.

Jake Bernstein: So, let me just start by asking, what do you mean by control performance?

Jared Pfost: And we're going to talk about that at length today, probably more than your listeners want to know. When we look at controls across our information security domain from data, and application, and devices, and networks, and response, et cetera, there's a number of things that we can measure, and we'll talk about what should we measure, and why, and how to communicate that.

Jake Bernstein: Okay. And if you had to provide a super high level overview of security metrics, maybe not from a technical standpoint, but how would you describe the mission here? The goal of the security nutrition? I just made that up, but I like that word.

Jared Pfost: Part of your question is, what should we measure? There's a number of controls across our landscape. And, if you're starting out from scratch, then leverage your risk assessment processes, like with cyber risk opportunities, to identify across our landscape, which controls should we have an investment discussion about? Where are we strong? Where are we weak? What should our priorities be in closing that gap? That risk assessment question will start to feed where we should start measuring.

Jake Bernstein: Interesting. So, Kip, maybe you want to dig into... You said you've been a CISO. And also, I'm curious, Kip, because you and I have talked about metrics before, but really... Maybe not metrics, but we've talked about FAIR, right? And I think we've been pretty honest on the podcast about our distrust of FAIR, which is not necessarily a FAIR way of saying it. It always has seemed like the juice isn't worth the squeeze. It's too expensive to do it. But I suspect that that is less and less true as we have more and more data. Back in the day, before SIEMs, and SOARs, and all the other four seemingly four letter acronym tools out there, there might not have been a ton of data. So maybe doing FAIR was impossible. But, I'm really curious, because I had an experience just the other day this week where another lawyer, he's just a general corporate lawyer, not a security guy at all, but he works with family offices, right, so rich people, and he was asking me, "What's a question I could ask my clients to determine if they need more security help?"

And, I've thought about that in the past, but I came up with something new this time, and it was a single question, and we both thought it would be really useful, which is, "What's your security budget?" And we don't really care if the answer is anything but zero. So, this was a binary metric that we were looking at, with a very simple goal of, "Are you even paying attention to this or not?" And we decided that if the response was zero, then that client is not paying enough attention to security. If it's almost anything but zero, then the assumption is, they've at least thought about it.

Jared Pfost: Yeah.

Jake Bernstein: But that's too basic. So what do you think?

Kip Boyle: Yeah, I agree too basic for the conversation that we're having. But, an interesting insight into how you can use just a binary one, zero measure to just begin to understand something, which is, "What do these people think about security?" And as far as FAIR is concerned, during the pre-show, we were talking with Jared, and he said something super insightful, which is, using my words, but know your audience. So, all I can tell you is, I've never been in a situation where I've had to use quantitative risk analysis tools like you get with FAIR, because my audiences have never asked me to do quantitative risk assessment. But I know that there are organizations out there that are using it and seem to be pleased with the results that they're getting. But, I think it requires a very large organization, because it's very resource intensive, and it requires special skills.

And so, it just hasn't come up for me. And so, I wouldn't know what to do with it, really. I suppose I could go learn it, but I'm not a statistician. I'm sure Jared has a lot to say about what I'm opining on it the moment. But, if we could not open up that can anymore.

Jared Pfost: All right, we'll put it away.

Kip Boyle: Yeah, because we don't want to talk about risk assessment so much as we want to talk about metrics.

Jake Bernstein: And that's what I'm... Yes, thank you. Because there is a difference, and we need to make sure we hit on that.

Kip Boyle: Yeah. Now, as a person who's worked as a chief information security officer, I got to tell you that I have metrics envy, because I would be sitting in a room with somebody from finance, somebody from operations, somebody from sales, and it just always seemed like they had one or two key metrics that they could share that would really summarize the overall situation that they could talk about, right? So, ROI, or maybe ROE, return on investment, return on equity, or salespeople could talk about conversion rate. And there was all these really super simple, yet useful, metrics. And I struggled so much with metrics envy, because I felt like I didn't have anything like that. And so, I struggled a lot. So, Jared, here's my question for you. How should people like me think about building up metrics that matter the most? I've never been able to find a one size fits all metric. Have you seen anything like that? Or is this really a bespoke situation, where everybody's got to figure out for themselves?

Jared Pfost: Firstly, I want to go back, I know you said you don't talk about everything there. I do have experience with FAIR. I do have experience with qualitative as well as hybrid. And in every modere's a part of the likelihood story that talks about control effectiveness. And that's what we're going to measure across these metrics. Now, to your last question, is there one control effective measurement that applies to everyone? Not necessarily. And that goes back to when you look across, you assess your risk, which areas of control are most impactful? And, for most folks, it's going to be around remediation management. For more mature areas that might be in preventative stack or responsive stack. That's where we usually start out is looking across our controls, how effective are we with remediation, and is our pace of remediation aligned with the risk tolerance of our business?

Kip Boyle: Okay. Now, let's take that a step further. So, Jared, can you tell us a story about a time when you actually had to go through this process and help people figure out what controls they should be measuring, and how they actually came up with the right measures?

Jared Pfost: And so, we've mentioned a few times about performing that risk assessment process to identify where our largest gaps, where do we need to make investment decisions? Then, another really helpful construct is... Earlier I mentioned, we know the types of things we want to avoid. Some folks haven't got there yet. So, what I've done in multiple organizations as I've built up GRC and measurement programs is developed a list of high level loss scenarios. So, these are the impacts we want to avoid, like application breach, exposing customer data, for example. We can go across our CIA triad on things we want to avoid. And each of those impacts is going to have an associated, and using the Mitre term, kill chain. So across all these steps, the potential vulnerabilities that could lead to this risk being realized, what are the controls that are minimizing that risk and controlling those vulnerabilities? And, across that kill chain, we can measure the effectiveness of each one of those controls.

So, for example, we can talk about this as we get into the actual metrics that I've seen work in the application space. So, from the prevent standpoint, how many of our internally developed applications are enrolled in secure developments? How well are we performing in remediation management? How well are we doing with training? Each one of those areas has a step in the kill chain, and we can measure where we're weak in that kill chain, we can then use that to say, "Is this control effectiveness appropriate for that loss scenario?" So there we've taken, why should the business care? All the way to how are we managing... Or what vulnerabilities are we trying to manage? To how well are we managing those today?

Kip Boyle: I love it.

Jake Bernstein: That's really powerful. I was thinking as you were talking, all of that is possible, but it's definitely a lot of work. And I was just reflecting on... Thinking about this as a maturity level, and I'm not using this in any official capacity, any actual maturity rating of any kind. I'm just thinking, I have so many clients for whom this level of analysis would be foreign. But yet, they could do it. It's just that it would be a lot of effort. And the funny thing is, particularly with what you were saying about using the Mitre kill chains, that doesn't require a bunch of super fancy technology at all. It almost is sitting down and just walking through these lists and thinking about what you have, which, I don't know about you guys, but sometimes I feel, that is one of the hardest things to ask people to do. A lot of the times, it's easier, "Oh, let me just buy this fancy blinky light box, and plug it in, and voila, I am secure." Even though, putting in the effort is what would really make them secure.

So, that's an observation I'm making only because once you explain it like that, Jared, it's like, "Oh yeah, that really is quite doable. Why don't more people do it?" I don't know.

Jared Pfost: You bring up a really good point. And I would argue, it's not that hard once you know what success looks like. However, there's an initial step to your point, is some organizations do not want to have that conversation. And I'm thinking back to the recent news about the SEC action and that'll probably be a whole other podcast that you're going to do.

Jake Bernstein: Oh, yeah. Kip, we haven't even talked about that.

Kip Boyle: I know. I've talked with customers about it, but not with you.

Jake Bernstein: Ladies and gentlemen, we'll save that for later. That will be a future podcast.

Kip Boyle: Yeah, it's on the editorial calendar. Thank you, Jared.

Jake Bernstein: Yes.

Jared Pfost: Yes, if you like this session, I would love to contribute my opinion on that one.

Jake Bernstein: Yeah, we may have you back for that. In fact, we could do a round table of security professionals and CISOs. But, we'll talk about that later. Yeah.

Jared Pfost: Your observation that this could be difficult, or take a lot of resources, and I was making the assertion, it's not that hard when you know what success looks like. It's really important for security practitioners to understand if their business leaders want to have this conversation. Many of them do not. Especially whether you have a measurement program or you're thinking of building one, highly recommend piloting the measurement program and pressure tests with your business leaders. One, do they want to measure current control effectiveness? And then, what I think is the fun part, do they want to have a discussion about what is the appropriate control performance?

Jake Bernstein: Why do you think that is? Both of you.

Kip Boyle: You beat me to the question. I was just about to ask that. Jared, why do you think people don't want to talk about it?

Jared Pfost: It's something that they hire the chief information security officer to worry about. "That CISO is going to help us manage our cyber risk. As a business leader, I have business objectives to accomplish." Someone who doesn't do security stuff.

Jake Bernstein: That's somebody who doesn't understand that cyber is a business risk, Kip.

Kip Boyle: Yeah. I think it's also because people want an easy button. And even as clear as Jared just described how to do it, that's not an easy button. That's a hard buttons.

Jake Bernstein: Do you get this? I mean, so I look at it maybe a little differently from the legal side. I almost wonder if it's a head in the sand defense. Right? Like, "If I don't know, then I can't be called to account for failure of things I didn't know about." Right? And, Kip, this goes back to one of my favorite episodes, the good old business judgment rule way back when from fiduciary duties, and corporate leadership rules, and things that are actually really, really important. And it would probably be great if more CISOs were fully aware of all of those laws and those realities, or just security people in general. But there is a component there of, "What I don't know can't hurt me." Which, of course, is fallacy. That's just completely wrong.

But, there's the temptation to go there. Because, I think Jared... And maybe it's time to start bringing in some specifics to help with the conversation, but it seems to me that if you come in and do your measurements, and actually get metrics, it becomes almost too easy for someone to say, "You're doing this well." Or, "You're doing this poorly and you should do it better." Right? And, the reason that you might not want to know that is that then a third-party can say, "Well, you knew you weren't doing that well enough." Right? And it's a little bit like the old Ford Pinto issue, right? Or tobacco, right? Did they or did they not know that cigarettes caused cancer? Did they or did they not know that the placement of the fuel tank would cause explosions and more deaths? Or did you not know that your MFA was ineffective? All of these things are the same.

Kip Boyle: I'm in a deposition now.

Jake Bernstein: Oh, yes you are. This is what happens when you have me on. But look, you guys, this is important stuff. Because, at the end of the day, if you want to drive corporate behavior, you need to have reasons that are not just grounded in because it's good for security. Because, they don't care what's good for "security." They care what's good for the company and their own hides, right? I mean, that's just the way people are. So, I mean, again, what we will have here-

Kip Boyle: Well, I mean executives don't get bonus because they have great security.

Jake Bernstein: ... No. And in fact, apparently, you can get sued criminally or civilly by government agencies if your security isn't good enough. It's a very interesting time. Just to be less opaque about it, we are talking about the SEC's recent charges against SolarWinds and the SolarWinds CISO. That gives you an idea of timing of when we recorded this. But yes, I do think it's timely to have this discussion. So Jared, I said a whole bunch of things. What do you think? What's your response?

Kip Boyle: To any of that.

Jake Bernstein: To any of it.

Jared Pfost: Usually, the head in the sand strategy comes from lawyers in my experience. So it's really refreshing, Jake, to hear a lawyer say, "That's a fallacy." Because there's also this driver of performing due care. And, I don't think the SEC is going to sue people for security that isn't good enough. They're going to sue... Or they are suing people for organizations that obscure their current performance. Due care is a process that even though your security may not be at your optimal level, if you have a program that understands where you're at today and you have a program to address it in the future, then Kip goes and gives his deposition. He's going to say, "This organization is applying due care to manage their risk at an acceptable level, given their business today. It's not perfect, but we have a plan to get there." For most CISOs that present to board of directors, that's almost the same story. "We're not there yet, but we have a plan to get there."

Kip Boyle: Right.

Jake Bernstein: Well, and Kip, you and I would say, it's a journey, not the destination's.

Kip Boyle: Yeah. That's right.

Jake Bernstein: You're never going to get there. And just to be super clear, even though this is not an episode about the SolarWinds SEC investigation or charges. On its face, the charges aren't, "You had bad security." The charges are, "You lied about the level of security that you had." And those are two very different things. Very, very different. And, metrics are so critical. You're right, Jared, there are lawyers who would say, "Don't measure, because then you can't be held to account, because you didn't know." Or you don't say anything. And that might've worked a decade ago, might've even worked five years ago. It's not going to work going forward. I think that's pretty clear.

Kip Boyle: Yeah. That's a strategy that's very long in the tooth as they say. And yeah, it's even arguable if it was even ever really a reasonable strategy. But, okay, so one other thing I want to say about Jared's approach is I really appreciate how consumable what you say is. In other words, I'm used to people talking about security metrics where they're using a very advanced vocabulary. It's very difficult to follow what they say. They get hung up way, way, way down in the weeds, and their motivations for the measurements that they want to make, I don't always understand what their motivations are. So I want to compliment you, Jared, for being able to talk about it in a way that just makes it clear. And, it makes it easy for me to understand. And, you're one of the few people that I know who can do this.

Jake Bernstein: And let's go ahead and specifically talk about it a little bit. We're 24 minutes in, we still got about 20 minutes, give or take. Jared, walk me through some metrics. What are we measuring? Are we measuring controls or are we measuring risk? Help me understand.

Jared Pfost: And thank you for that question. And the answer is yes. As we talked about earlier, we're going to be measuring control performance, and those controls are going to be aligned to loss scenarios and that Mitre kill chain. So, by measuring their control effectiveness, we are participating in measuring that risk. And, earlier we talked about how do we prioritize what we measure? We're going to use our risk assessment process, our incidents, and identifying what are those key controls that we need to have an investment discussion about. And, that's the difference for me between a key risk indicator and just a metric. A key risk indicator should be driving a decision. Do we need more or less? Should we go faster or slower? Once we've reached acceptable performance for this control, move that metric into your operational scorecard, it's no longer a key risk indicator. So, that's my pressure test. When I'm refreshing my list of KRIs, is this measure driving a decision with my leadership? If not, let's get it out of here. It has to be adding value.

Jake Bernstein: That's really good.

Kip Boyle: I love the way you contextualize that. How do you know you're getting business value from your metrics. I really appreciate that you made that connection.

Jake Bernstein: And I'm trying to make sure I understand. So, a key risk indicator, that is when... Maybe try to say that again. I really want to understand what a key risk indicator is, versus a... So, is a metric a key risk indicator in waiting, or is a key risk indicator of a past metric?

Jared Pfost: And, I don't want to over-complicate things. The spirit of what I was saying there is that when we look at our key risk indicator dashboard, everything on that should be driving a decision.

Jake Bernstein: Okay, I see. So it's more of a key risk indicator, maybe the key is the key, right? It's not a key if it's no longer driving a decision, therefore it can leave the key risk indicator dashboard. Got you. That helps. Okay. Let's get a little specific. What are some key metrics that people should look at? I am sure there are listeners right now who are just saying, "Get to the key metrics." What should we be looking at?

Jared Pfost: So, let's go there.

Kip Boyle: And that reflects, right, what I told you, which is, I've had this appetite for, "What are my key metrics?" Right? With this assumption that there are metrics everybody should have, right? And that it's not just about my unique situation. So, I want to hear the answer, Jared.

Jared Pfost: Yes. And I'm anxious to share the core metrics that have provided value for me in the past. And, we will run out of time, so I'll provide a list of these measures for those listening on the website. Next week, I'm talking at a security conference, so maybe we can link to that presentation if it's helpful. But, to go back to what we talked about earlier, the area where most organizations struggle is remediation performance. And, is that remediation performance aligned with the risk tolerance of the business? So, start there. Let's say that the year remediation performance is optimal. Then, now let's talk about some of the other metrics. And earlier I rattled off some typical domains as far as application, device, cloud configuration, inaudible access management, detection response, GRC, et cetera.

Each of those categories, I have a handful of metrics that inform what is our current control posture within those areas. So we have very tactical control performance metric, and each of those can be rolled up to those higher level categories. So, there's those five categories that I mentioned. And then, we can have a more strategic conversation or a board level conversation of, "We're not there yet, but we have a plan to get there." And, we have that tactical data to fall back on and these are the specifics that we'll talk about next.

Kip Boyle: Okay. So, one thing I heard you say is that a common area of struggle for organizations is remediation. And by that you mean, applying security updates, changing configurations so that systems are more secure. Am I following you?

Jared Pfost: Correct.

Kip Boyle: Okay.

Jake Bernstein: So, a concrete example of that would be the average amount of time between the release of a patch, and your actually patching it. Is that one example of a metric that you might look at?

Jared Pfost: It's not an example I would use, because of the flaw of averages, but it's close. So, for example, say, the percentage of your application vulnerabilities that are mitigated within your policy timeframe. So, for example, internet facing, critical vulnerability, how many days should that be fixed based on your risk posture? Some organizations, it's 7. Some it's 3. Some it's 30.

Jake Bernstein: So the measurement there is, if it's 3, of all of them, how many were fixed within three days?

Jared Pfost: Correct.

Jake Bernstein: Okay. I really want to ask you about the flaw of averages, but I'm not going to right now. You knew it. Kip knew it. He could see me locked up when Jared said that.

Kip Boyle: That's probably the first thing Jared said where I felt like I needed to ask for him to explain it, because Jared, you generally don't use words or terms that I don't already understand. But we don't have time to unpack it right now, which is too bad. But, I wanted to ask a follow-up question, which is, in order to have metrics, and I think that one's a very insightful one, you need data, right?

Jake Bernstein: And policies.

Kip Boyle: Well, yeah. So, you need a standard against which to measure, but then, you need data that reflects what you're actually doing.

Jared Pfost: Correct. And getting that data can be expensive, depending on how global and complex your environment is. And that goes back to my story about it's important to pilot. So, before the security team is being tasked to build this measurement program, is it actually going to provide value, and wanted, and needed? So, one area that's fairly simple to collect, going back to remediation management, is through our vulnerability scanners and our assessments. There's usually plenty of findings. Those findings are going to be prioritized by severity. And then, we can measure the start and stop date for each of those. That's a really good data source. It's not expensive to get. And you can pilot that process. Say, we have a policy that says we need to mitigate our vulnerabilities. More mature organizations will have gone through that risk management discussion with the business of saying, "How fast should we fix our critical vulnerabilities on these types of assets? So that's our target, and thus we have our current control versus our target performance."

Kip Boyle: And I think, a key thing that you said in there is that sometimes it can be expensive to collect the data that you need, or even logistically, impossible. So, if you've got a far-flung enterprise, and you're trying to get data from Asia, and you're based in Europe or North America, it could be extremely difficult or impractical, really, to get data that's as fresh and as relevant as you might be able to get in other areas of your enterprise. Have you run into things like that, Jared? I mean, I have.

Jared Pfost: Many, many times.

Kip Boyle: Yeah. Yeah, it can be very difficult.

Jared Pfost: It's important to separate the process difficulty from the technology. The one CISO I worked for, he decided, yes, he wanted to have a data-driven program. And so, I gave him my list of metrics. "This is what's worked in the past." And he said, "Build the process manually before I'm going to give you millions of dollars to automate the collection enrichments and visualization of this data."

Kip Boyle: That seems like a smart strategy. Keep talking.

Jared Pfost: And so, that's what we did. So, pilot those measures, do it manually with an Excel, pressure test that process of when we see our actual performance and it's not meeting what our target performance is, is that measure of driving a decision or is it going to be ignored? If that's driving a decision, you can use that little victory to justify additional investment.

Kip Boyle: Right.

Jared Pfost: A quick one success story, one of my highlights of my career is working for a financial services organization and the CISO said, "Yes, I want a data-driven program." I signed up and we collected all this information from a data lake, and then to a relational store, and then we used a visualization tool. And the CIO saw the security performance across her four business divisions for the first time and said, "This is not acceptable. I'm going to make security one of my top five priorities for the year." And, real change happened.

Kip Boyle: That is fascinating.

Jake Bernstein: So, I'm cheating, I have the same reference document that you provided us, and I'm going to just look at one of these that says, "Percent of assets meeting authentication requirements, EGMFA, and PAM." So, walk us through how you could measure that, and then have it drive decisions.

Jared Pfost: For that specific measure?

Jake Bernstein: That specific measure. Yeah.

Kip Boyle: Yes, please.

Jared Pfost: And also, to take a step back, earlier I mentioned those five domain categories. One of those was a data access management, and there's a handful of metrics within that domain. So, this would be one of those measures. And, especially around IAM, VAX is going to determine what is your degree of automation to be able to get that information. So, sometimes that's harder around the IM space. So, I believe this one was percents of assets with inaudible at the top. So then, of course, you know what's your denominator, what is your population of assets? And then, because that's a large population, they have an automated way to understand what level of either privileged access management or multifactor authentication for interactive accounts versus non-interactive accounts exist. So, you picked a really good one to say that you need a degree of sophistication to measure performance around that control. That's much harder than, "Am I fixing my vulnerabilities for my scanner on time?"

Jake Bernstein: But let me dig into this, because there's little wheels that are turning in my head when we talk about this. Let's say that we measured this, let's just say we could, and let's say that the first time we measure it, we get 25%. So, 25% of our assets are meeting our authentication requirements. What does that actually mean and how would that drive a decision? And, let me keep going, and then you can tell me if I'm right or wrong. Well, if I'm that organization's cybersecurity lawyer, I'm going to say, "Guys, 75% of your assets are not meeting your own authentication requirements. Something bad is going to happen, it's just a matter of time. You need to fix that." And I would expect them to be like, "Wow, you're right. I guess we'll fix that." And spend money, take action, do whatever. And I don't know. And I guess a question is, when do you stop? 100% seems unachievable in a way, but what is the goal?

Jared Pfost: And, excellent, thank you for bringing this one up. And that's the fun part, what is acceptable performance while optimizing costs? And that's why I enjoy working with business leaders and non-technical folks to facilitate this conversation is ultimately they're accountable for their performance. So yes, 100% isn't going to be achievable in a dynamic, elastic computing environment. 25%, it may be acceptable, may not be. It goes back to our loss scenarios, and our risks, and our kill chain. And where does authentication factor in that kill chain? So, we look at from the controls within, say, a breach and loss of customer data or disruption due to ransomware or malware, where does authentication play in that kill chain? And, it's very prevalent. And when you look at your assessments, and your pen test, your red team activities, the industry vulnerabilities, when you go back to the Verizon data breach report, you can see where authentication plays in that kill chain in realizing those risks.

Jake Bernstein: Kip, he's changing my mind on measuring things.

Kip Boyle: Well, good, I'm so glad to hear you say that, because that is really why I thought Jared would make a great guest, because for anybody out there that's struggling with metrics and they're just not pulling something together that's driving decisions, that's creating business value, this is exactly where we wanted to get.

Jake Bernstein: You can probably tell, I'm getting very excited over here. And so, Jared, I want you to finish your thought in just a second on how you end up at a goal. I understand that it's going to vary. There may be a situation where 25% is acceptable. I'm guessing, with this particular example, most of the time, it's probably 90 plus. Maybe it even looks more like an SLA, where it's like, is it 99 or 99.9? I don't know. You can opine on that. But what I want to point out is there are very few areas of business where it's a given that the executives will want metrics, right? Nobody is going to run a sophisticated business without financial metrics or sales metrics. Why do so many think it's okay to run a business without security metrics?

I think, Kip, that this is one of those things that changes the standard of care, that changes what it means to be reasonable with your security program. And, maybe not today, at least in certain places, but certainly in the not too distant future, and if I was a prosecutor, or a regulator, or a plaintiff's lawyer, I would make the argument that if you're not doing metrics, you are not being reasonable. Ergo, your security program is not going to get you off the hook for fines, penalties, et cetera.

Kip Boyle: But, would you say we're at that point now?

Jake Bernstein: I think, for many companies, yes. I'm not going to say for all, because as we said, it does depend, right?

Kip Boyle: Right.

Jake Bernstein: This could be a hard one to measure.

Kip Boyle: Which is your favorite thing to say.

Jake Bernstein: Yes.

Kip Boyle: So, we know from the Federal Trade Commission that reasonable cybersecurity is dependent upon the types of data that you collect, and also, it depends on your size and sophistication. Right? So, there's some criteria here. When you told me just now that you thought that for some organizations these metrics are needed today, and that it's per se unreasonable not to have them, did you pass it through the FTC filter when you thought about that?

Jake Bernstein: Okay, I pass everything through the FTC filter. I can't not do that. It's so ingrained in how I think.

Kip Boyle: I just wanted to make sure.

Jake Bernstein: And that's really what I was thinking of. When I said it's going to be reasonable in some situations, if you've got the infrastructure to do the measuring and it's only going to cost you an incremental amount to actually do it, then you probably ought to be doing it. If you're in a situation where you'd have to spend three times your entire annual security budget to even begin to measure it, that probably isn't yet reasonable to require you to do it.

Kip Boyle: Okay. Okay. Well, look, we're about 45 minutes. We're coming up 45 minutes in the episode. We could easily continue. We could have a part two and a part three. But, I want to see if we can wrap it up right now. And I just want to start that process by thanking Jared.

Jake Bernstein: Before we fully wrap it up, Kip, we're only at 43 minutes. I want Jared to react to what we just said, because it feels like he really wants to.

Kip Boyle: Okay.

Jared Pfost: Thank you. And, two points. One, I want to go back to your tactical question, but also to your larger area, is measurement required as part of demonstrating due care? Back in the day, when the new cybersecurity framework was first open for public opinion, I just had one piece of feedback. Of course, I had a bunch, but I only shared one, and that is, mandate security metrics in a nutshell. Of course, it didn't get in there. Now, with the 2.0 release coming up, and now we have the govern section, it's one step closer of requiring a measurement program, especially with the new SEC guidance of being able to demonstrate your risk management program. How are you going to demonstrate that? And wisely, they don't say how. I would argue that having a measurement program, no matter how rudimentary or advanced, is a great way to demonstrate that program.

Jake Bernstein: It is. And, just to make you even more excited, Jared, the way that the legal system works, it doesn't have to be written down in a law for it to become the standard of care. It will become the standard of care if it is the standard, or should be the standard of care, because someone who wants to defend themselves will do it, and then it will be the standard of care for everyone else. Because they did it, why can't you do it? Oh, you could, you just didn't.

Jared Pfost: Right? Okay, so now let's get really tactical back to your identity access management, your authentication metric. And say, you're at 25%, now you're having the fun conversation of what is acceptable. So, each one of those metrics, in my experience, it benefits from a short-term and a long-term target. The long-term is ideal. So say, you have conversations and your CIO and your CTO said, "Yes, based off on our business and our growth rate, we want to get a level of maturity where we're at 90%. That's long-term. For this year. Maybe we want to move that 25%. We're going to invest in some automation and get that up to 50." You're still not at that long-term goal, but you have an reasonable, achievable milestone where you're demonstrating due care.

Jake Bernstein: Which would matter in a regulatory or a legal proceeding.

Jared Pfost: Exactly.

Jake Bernstein: Someone could say, "Oh, you were only at 50%." But the response would be, "Yes, but we doubled our effectiveness from the previous year. We're getting to 90."

Jared Pfost: Yes. Victory.

Jake Bernstein: Yep. Very good. Okay, Kip, now you can wrap it up.

Kip Boyle: Okay. Okay. Okay. Thank you. No, I was not trying to shortchange Jared or create tension with the audience, because I was not going to let him react to the question, but-

Jake Bernstein: Not the audience, Kip, me. I was the one. I wanted the answer, man.

Kip Boyle: ... Okay, I'm just going to say one more thing and then we're going to wrap the show up, which is, I hope everybody recognized that even though we are talking about metrics, which are numbers, notice how much of the conversation was around people. The metrics that don't drive decisions aren't that valuable. Metrics that don't create change in the organization for the better are not that valuable. That's all about people. It is all about people. So, don't get hung up on this idea.

Jake Bernstein: And risk management people, what's the name of this podcast? Cyber Risk Management.

Kip Boyle: That's right. Cyber Risk Management Podcast. So, okay, that does wrap up this episode of the Cyber Risk Management Podcast. And, today we explored how you can use security metrics to measure outcomes and help leadership make decisions, help people. And we did that with our guest, his name's Jared Pfost, and I'm so glad he was here. Thank you, Jared. We'll see you next time.

Jared Pfost: See you next time.

Speaker 1: Thanks for joining us today on the Cyber Risk Management Podcast. If you need to overcome a cybersecurity hurdle that's keeping you from growing your business profitably, then please visit us at cr-map.com. Thanks for tuning in. See you next time.

Headshot of Kip BoyleYOUR HOST:

Kip Boyle
Cyber Risk Opportunities

Kip Boyle is a 20-year information security expert and is the founder and CEO of Cyber Risk Opportunities. He is a former Chief Information Security Officer for both technology and financial services companies and was a cyber-security consultant at Stanford Research Institute (SRI).

YOUR CO-HOST:

Jake Bernstein
K&L Gates LLC

Jake Bernstein, an attorney and Certified Information Systems Security Professional (CISSP) who practices extensively in cybersecurity and privacy as both a counselor and litigator.