Speaker 1: Welcome to the Cyber Risk Management Podcast. Our mission is to help executives thrive as cyber risk managers. Your hosts are Kip Boyle, Virtual Chief Information Security Officer at Cyber Risk Opportunities, and Jake Bernstein, partner at the law firm of K&L Gates. Visit them at cr-map.com and klgates.com.

Jake Bernstein: So Kip, what are we going to talk about today on episode 174 of the Cyber Risk Management Podcast?

Kip Boyle: You convinced me we couldn't put it off forever. So we're going to finally talk about the massive CrowdStrike IT outage, the great outage of 2024. Now, it's been, gosh, what? 90 days at least since this happened at the time that we are recording this. By the time this episode's released, it'll be probably another 60 to 90 days on top of that. So we're late. But that's okay because inaudible.

Jake Bernstein: We're not a news program. As we've said before, we're not a news program, and we wanted to have the ability to do more of a postmortem of this event.

Kip Boyle: Absolutely.

Jake Bernstein: So take it away, Kip. Let's start on this.

Kip Boyle: Yeah. All right. So we'll begin at the beginning. Now, you told me that you had access through your law firm to a really good version of Copilot, which I don't. So it was really cool to see you bring this in here, a summary from Copilot, which of course, was able to go out to the internet, do an internet search, and then collate and-

Jake Bernstein: Actually, Kip, I've a secret to tell you, you actually do have access to that version of Copilot because that's just the copilot.microsoft.com version.

Kip Boyle: Well, I don't use it.

Jake Bernstein: There you go.

Kip Boyle: So I don't think I have access to it.

Jake Bernstein: You should try it. It's good. They're all interesting.

Kip Boyle: I've been dabbling with so many. I'm using the API to talk to Claude 3.5 Sonnet at Anthropic. I also have an API that lets me talk directly to any of the ChatGPT models. I've been using Fabric, I've gotten a premium subscriber of ChatGPT, the consumer interface. I'm all over the place, but Copilot is just not one of them.

Jake Bernstein: And that's fair. There's too many.

Kip Boyle: Yeah, there's a lot. But I think this is good. Okay. So let's just see what Copilot said. So it said, "On July 19th, 2024, a faulty update to CrowdStrike's Falcon platform..." So Falcon is... I hear a lot of people say, "Well, we use CrowdStrike." Okay. Well, that's fine. That's the company name. Really there was a product that they sell called Falcon, that's their EDR, and that's the one that is the source of this massive IT outage and had to look at blue screens of death all over the world. There was at least 8.5 million Windows devices did this.

Jake Bernstein: 8.5 million blue screens of death. That's got to be a record.

Kip Boyle: Yeah. Well, I think it is. It's historic. And I think the other thing that's really historic, because I think looking at 8.5 million blue screens of death is interesting. But what I think is notable is that the outage disrupted critical services across a vast swath of sectors, including global air travel, healthcare, like patients had to put off elective surgeries, and the business operations of so many organizations. So that was the incident. The recovery process was fascinating as well because it took a lot of effort from CrowdStrike, but also Microsoft, that they had to work together to do the service restorations. There was also some Azure problems that were unrelated, but crept into this, which made it even more challenging. And CrowdStrike CEO, George Kurtz. Now, he stated that he believes that ultimately this incident made his company stronger, and improved their relationships with their partners. I believe that, but I wouldn't say he probably thought that that was the best way to do it.

Jake Bernstein: For sure, not.

Kip Boyle: I mean, he is making lemonade here.

Jake Bernstein: Yeah, he is.

Kip Boyle: Big jars of it. So recovery wasn't easy. And I think as far as lessons learned, so if we go into our post-mortem here, this outage has highlighted the importance of robust disaster recovery plans.

Jake Bernstein: Has it?

Kip Boyle: Comprehensive update management and enhanced resilience for IT infrastructure everywhere. And that's in the big picture. Now, in the small picture, I think people rightly asked, why in the world is Falcon, and it's not the only one that does this, but why is Falcon allowed to interact with Windows at the kernel level for its operations?

Jake Bernstein: And Kip, I'm glad you asked that. It's almost like I put it in the script.

Kip Boyle: Almost? Good thing I stick to the script, Jake.

Jake Bernstein: It's a good thing. Because I do want to quickly touch on this. So first of all, Kip, were Apple devices affected by this?

Kip Boyle: I don't believe they were.

Jake Bernstein: No, they weren't. And why not? Well, it's because, one, I'm not honestly sure if there's even, there's even a Falcon product for macOS.

Kip Boyle: There is?

Jake Bernstein: I believe that there is.

Kip Boyle: And for Linux.

Jake Bernstein: But the difference is the access to the kernel operating system. And for those who don't know, Kip, what is the kernel? Just a high level.

Kip Boyle: It's so funny. So you want me to describe something that's deep, deep, deep down at a high level? Okay, I can do that.

Jake Bernstein: I do, yes. Abstraction.

Kip Boyle: So the kernel, spelled with a K, K-E-R-N-E-L, if you don't know much about operating system architecture, this is like the heart of the system. And every operating system pretty much has this. There's different ways that you can architect them and so on and so forth. But leaving all that aside, just think of the kernel as the beating heart of the operating system. The part that has complete control and access to everything, every process, every piece of data, every device. The kernel is root. It's even more powerful than root, but just think of it as the most powerful place in an operating system.

Jake Bernstein: Got it. Okay. So on Windows, Falcon directly hooks into the kernel?

Kip Boyle: That's right.

Jake Bernstein: And that means what?

Kip Boyle: Well, that means that Falcon runs with the highest level of privilege possible on the machine. It runs at a higher level of privilege than human beings do. So that means that if Falcon, or really any, Falcon is not the only one here, but anything that can hook directly into the kernel, can bring the system down. I mean, there's no protections. It's only by the grace of it's good behavior that it doesn't bring your system down.

Jake Bernstein: Got it. So somewhat famously, over the last five years, macOS has kicked everything that's everything out of the kernel, essentially. There used to be system extensions that I think accessed the older versions of macOS kernel but that's impossible. And we're not going to spend a lot of time on it. But I wanted to just point out that there was actually 2009 antitrust agreement between Microsoft and the European Union that at least Microsoft has said that this agreement forced them to sustain low-level kernel access to 3rd party developers. What they're saying, again-

Kip Boyle: It's throw you under the bus man. inaudible profession in a way.

Jake Bernstein: In a way, yes. Or the regulators, however you want to look at it. But the bottom line is what they're saying is that, look, we have to continue allowing, or we had to allow third parties access to the kernel at the lowest level.

Kip Boyle: You made us do it.

Jake Bernstein: You made us do it. So whether or not that's true, it doesn't really matter, we're not going to focus on it.

Kip Boyle: But that was their offered just because for why this practice has allowed.

Jake Bernstein: It was, yeah. The EU pushed back. macOS since 2019, like I said, has limited access to the kernel. Linux I don't think there's some other endpoint security framework thing that you can use. Anyway, that's just why the technical underpinning of this.

Kip Boyle: Why was the devastation so tremendous on these computers effective?

Jake Bernstein: Yeah. And then let's also be clear, Kip, was this a cyber attack?

Kip Boyle: Well, later on in the outline for the episode, you asked me to describe my top lessons from this. And I think one of the top lessons is it's so easy to shoot ourselves in the foot these days in a very, very big way. And I bring this up now because if you say, was it a cyber attack? Well, in a way, they cyber attacked themselves, didn't they?

Jake Bernstein: They did.

Kip Boyle: It wasn't the result of a malicious outsider. It wasn't like M.E.Doc with NotPetya where some outside party slipped in a malicious update. They did it to themselves. So if you're Delta Air Lines, you might call this a cyber attack.

Jake Bernstein: Obviously, we think about the Verizon data breach investigations report, and it wouldn't be called a breach because there was no confirmed disclosure or acquisition of information. But it would absolutely 100% be an incident.

Kip Boyle: Yeah. It could be a breach. It could be breached from a GDPR perspective. That's something I've been hearing people talk about, but I don't think that's been determined. Keep going.

Jake Bernstein: So ignoring that for the moment, it would be an incident and it would fall under mistake. It would fall under error. And that's fundamentally what it was. And since we haven't actually said it yet, the actual disruption was caused by a faulty update to the Falcon sensor, I guess component of CrowdStrike Falcon. And I don't know that it's important for us to try to find the exact exact details, but basically it crashed the kernel, which caused the blue screen of death. That's what it did. It basically just crashed the computer.

Kip Boyle: Yeah. And it crashed it really hard too. That's the other thing here, is that in the worst cases, the crash was so complete that it took Herculean effort on a machine by machine basis to restore.

Jake Bernstein: Yeah, it did.

Kip Boyle: You had to touch the machine. You couldn't use a remote management tool.

Jake Bernstein: Well, because you couldn't get in. There was no boot, there was nothing to remote. I mean, that's a really important point.

Kip Boyle: It was totally brain-dead.

Jake Bernstein: There was nothing to remote access because there was no computer to access. It was unconscious on the floor, zombie. Okay. You're right. I did want to get to your top three to four lessons here. I just wanted to say one thing beforehand though, which was some of the articles I've been reading about this humorously, or maybe not so humorously, compare this to the Y2K panic. Wow, that is a phrase I haven't uttered in a long time. But for those of us who are old enough, and I remember, I mean, I graduated high school in 1999. There you go. I just gave away some personal information, but I remember this Y2K thing was going to be this. There were some people who were convinced that when the clock ticked over to the year 2000, that it was going to crash the world. We were all going to go back to the stones age.

Kip Boyle: Planes would fall out of the sky.

Jake Bernstein: Literally. I mean, there was legitimate concern. When I say legitimate concern, I mean people would otherwise go off and wear tinfoil hats, believed that there would be a major issue.

Kip Boyle: Yeah. It seen as a credible threat. Yeah.

Jake Bernstein: Yeah. There wasn't. I don't actually know for sure if anything happened, but it clearly was not-

Kip Boyle: Some things happened.

Jake Bernstein: Some things did happen.

Kip Boyle: I remember I got an embossed card, a membership card for something. Looked like a credit card, but it's just a membership card. And they didn't boss the expiration date on there, and it was like 1901. Because their system couldn't catch the rollover. So there were a few things like that, but nothing really, really major. And it's funny because some people think, "That was a whole lot about nothing." And I say to myself, "Well, thank goodness it turned out that way because a lot of people worked really hard to make sure that something that happen."

Jake Bernstein: See, and that's what gets missed is that it wasn't that... I mean, there was a lot of work put into making sure that the Y2K thing wasn't going to blow up everything.

Kip Boyle: Yeah. And it succeeded. Now, this is another example of how it's difficult to describe the benefits of assurance in systems and security because when it does its job, it's invisible and you have no idea that anything ever happens. So I think of Y2K as a success.

Jake Bernstein: It was a success.

Kip Boyle: Yeah. But most people who went through it would say that it was all, it was-

Jake Bernstein: It was a big nothing.

Kip Boyle: It was a lot about nothing.

Jake Bernstein: Which is the wrong lesson. Okay. So Y2K hyperbole aside, the CrowdStrike outage was a big deal.

Kip Boyle: It was.

Jake Bernstein: So Kip, here you go. You already hinted at one of them, but what do you think the top lessons are of this outage?

Kip Boyle: Well, I think the number one lesson is something that we talk about all the time. Cyber has to be managed as a material business risk. It has to.

Jake Bernstein: Yes, it does.

Kip Boyle: Now, a cyber incident or a breach, whether it rises to the level of materiality as defined by the Securities and Exchange Commission is not what I'm talking about. I'm just simply saying that a cyber error, in this case, a cyber failure also in this case can lead to material consequences. And we're talking about no less than $10 billion of aggregate damage that was created as a result of this. And that's just an estimate. But I think that's believable. Go talk to Delta Air Lines. Half a billion dollars of damage happened just there. And only a small fraction of this damage was actually coverable by insurance. So that's another thing that we need to pay attention to, is we've got insurance for all kinds of things, errors and omissions, general liability, all kinds of things. So to the extent that companies haven't thought about, well, we need cyber insurance, I take that as evidence that they're not managed cyber as a material business risk.

Jake Bernstein: Yeah, very true.

Kip Boyle: I think that's a big takeaway. I think the other big takeaway is the brittleness, how easy it is for infrastructure to just cascade into failure like this. I mean, you don't see other big pieces of infrastructure do this. Once in a generation, the electrical grid will do this, but not often.

Jake Bernstein: Not often. No. I've never seen that happen really. And oftentimes, the causes are different. They're more complex than... I mean, in a way, Kip, you were talking about, is this a cyber attack? And in one, I would say in the most colloquial sense of the term for the man on the street, no. It was not a "cyber attack." However, I think everyone can agree it would've made a hell of a good one.

Kip Boyle: If it was. If it had been. And the truth of the matter is we don't know. I mean, George Kurtz said it wasn't. Okay, I believe George Kurtz, but would he have any incentive to lie? I think there's a whole other conversation we could have, which I don't think we should now about, well, maybe it was a cyber attack and it was just covered up. I mean, I'm not really a conspiracy theorist, but I think you're right. This would've made a really great cyber attack. And if nothing else, it's a wonderful demonstration of what's possible when the cyber attackers really do get serious.

Jake Bernstein: It is.

Kip Boyle: Okay. So I've got two right now. You got to manage cyber as a measure of risk.

Jake Bernstein: All right. You get one more.

Kip Boyle: It's easy to shoot ourselves in the foot in a big way. And the third is, and I just couldn't help to say this, it's a fantastic way to win the Most Epic Fail award at DEF CON, which happened. George Kurtz on behalf of CrowdStrike accepted the Pwnie Award for Most Epic Fail at DEF CON in 2024. And good on him for having the lightheartedness to actually show up and do, and be there.

Jake Bernstein: I think you got to at this point.

Kip Boyle: I think if you're anybody, you do, but certainly not everybody in the business world would've shown up.

Jake Bernstein: I agree. No, that's true.

Kip Boyle: So kudos to him for being willing to do that. And I think it says a really wonderful thing about his character, and the sincerity of how he handled the situation.

Jake Bernstein: And we'll talk about a little bit later, like the stock price and things like that. But I do wanted to say that overall, I think CrowdStrike handled it pretty well.

Kip Boyle: Yeah. Considered, I couldn't think of a better way other than-

Jake Bernstein: No, I mean, I really they did. I guess I just want to be clear, this episode ultimately isn't about knocking CrowdStrike or faulting Microsoft for the access. It isn't about that at all. What it is about though is what I want to talk about for a little bit, which is how this highlights what the Wikipedia article calls centralization and homogeneity, which I'm calling the risk of a single point of failure and the risks of reliance on a small number of big players in the IT world. And like I said, I'm not singling out CrowdStrike. The entire shift to the cloud-

Kip Boyle: That's some great data point.

Jake Bernstein: Yeah. It's a data point. The entire shift to the cloud has been one big experiment in reducing the resiliency of the internet. And how ironic is that, Kip?

Kip Boyle: That's exactly the word I was thinking.

Jake Bernstein: Because the entire friggin point of the internet when it was DARPA net was redundancy and resiliency-

Kip Boyle: Survivability.

Jake Bernstein: Survivability in a nuclear war and literally, eliminating single points of failure.

Kip Boyle: Famously people said the internet routes around failure.

Jake Bernstein: Yes. Well?

Kip Boyle: Not this time.

Jake Bernstein: Not this time. So we've gone from millions of individually managed servers to these enormous data centers, often owned by one company. And yes, and this is all, by the way, just so everyone knows, this is all me just pontificating. I don't have a source for this part of the conversation. The source is me. Virtualization technology I think is responsible for a big part of the cloud migration, I mean just technologically. But in defense of the cloud migration and a lot of this, it does make economic sense to concentrate. And I mean this as a specific phrase, the "business of providing server infrastructure" to companies who are going to handle it at scale. This is the so-called hyper scalers. And why? Because economies of scale do matter. They're valuable. They create efficiencies, but they also create risks. And I think that the CrowdStrike outage could not have highlighted those risks better. So what do you think about that? What do you think of my pontificating?

Kip Boyle: Yeah. Well, I think the irony again is like, well, let's go to the cloud because it'll be more resilient, but then we find out that it actually has its own soft underbelly. There's no such thing as perfect resilience. So I'll bet a lot of outages have been avoided because companies have built their products and services on cloud that if they hadn't, that they probably would've been underfunded and they would've had creaky infrastructures. There would've been a lot more individual outages scattered all over the internet. So I think it's good. I think cloud is good, but anybody who puts too much stock in cloud, whether cloud is more secure from a confidentiality perspective or in this case, availability perspective, I think they do it at their own peril.

Jake Bernstein: Yeah, they do. And the idea of we need redundancy. This goes back to what you said early on about you need to have strong disaster recovery plans. You need-

Kip Boyle: And business continuity.

Jake Bernstein: And business continuity, you need to practice those. You never know what crazy thing is going to happen in the world. And if you want to continue operating, you have to be ready for it. CrowdStrike, I don't think it's going to lose all that much business. I don't think we have numbers on that at this point, but it doesn't seem like they've lost that much business. Yes, some people have left. That's going to happen. There's always customer churn.

Kip Boyle: And that's what we find from a lot of the reports that we reference, is that, when you have a breach or an incident that's big, you're going to get abnormal customer churn on top of your normal customer churn.

Jake Bernstein: Yeah. Okay. So Kip, I put something else in here, another Copilot thing. Why don't you take us through this other question prompt and response to continue on.

Kip Boyle: Yeah, sure. So this is where you asked Copilot to provide commentary on the historical significance of the outage. And then you've just got some excerpts here, which I think is really good. So it's considered to be the largest IT outage in history or one of, I'd have a hard time figuring out a bigger one.

Jake Bernstein: Yeah. I think almost everyone is saying the, or I mean, definitely one of, but I think-

Kip Boyle: It's the big one.

Jake Bernstein: It's the big one. It's hard to find one that's bigger.

Kip Boyle: The only other outage that I've ever heard that caused this much damage was NotPetya, maybe WannaCry, but this is top three. This has got to be one of the big three.

Jake Bernstein: Yeah, for sure.

Kip Boyle: 8 1/2 million devices disrupted globally, Windows devices, airlines, hospitals, financial institutions. I mean, this is a case study of critical infrastructure, and it's fascinating that we don't just have one part of critical infrastructure affected here. It zipped through a whole bunch of them.

Jake Bernstein: I mean, here you go, Kip

Kip Boyle: Cut right through them.

Jake Bernstein: Across the globe, 5,078 air flights, almost 5% of all global air traffic that day were canceled.

Kip Boyle: That's right. And that's just an air traffic. And then there's financial, there's hospitals. I mean, it just cut us off of the knees. 5.4 billion for Fortune 500 companies alone is the estimated costs of dealing with it. Big number. And of course, the aggregate cost is much higher.

Jake Bernstein: Is unknowable at the moment. It may never-

Kip Boyle: Yeah. No. I don't think anyone's going to get a definitive number, but there was some calculation that was done and floated around about, I think $10 trillion was the estimated global impact of this to everybody everywhere, all at once kind of thing. Anyway, it's a big number. So let's see here. The whole trust that we have and how software updates are deployed and managed, I think has been shook. We're completely revisiting the way that's being done. And I think that's actually very good. I remember when Windows updates first were announced that Microsoft was going to just auto-patch people. And I thought, "I don't like that." I don't feel comfortable that a company can just shove software into my computer whenever they think it's a good idea. And I have no control over that. That made me-

Jake Bernstein: And that's Microsoft, that's the operating system. Now imagine, now imagine, Kip, that you're allowing third parties who have access to the kernel to just shove it. Now, I will say it has changed. CrowdStrike has come out and they've basically said, "Okay. We're giving people more control over this. We're doing it."

Kip Boyle: And they're moving out of the kernel. I think-

Jake Bernstein: They are, yes.

Kip Boyle: ... they're access via API. So even if the exact same thing happens again, they'll be-

Jake Bernstein: It won't be more resilient.

Kip Boyle: ... resilient.

Jake Bernstein: Yeah, way more resilient.

Kip Boyle: Yeah. There's a lot of policy and regulation consequences. Congress is trying to figure out, do we need better policies? Do we need funding for crisis management?

Jake Bernstein: Also antitrust? I mean, there's a lot of people asking if the vulnerability that we were talking about related to single points of failure and homogeneity is an antitrust problem, and that affects a lot of companies. But again, it's hard because antitrust is not... I don't know, Kip. Sometimes my mind goes to biodiversity in this concept of monoculture. I think I talked about this at a panel when it happened, but the Irish potato famine, that was largely the result of monoculture, which means that all the potatoes in Ireland were of-

Kip Boyle: The same strain.

Jake Bernstein: ... the same strain. They were all related. And a single, I think, I don't remember. Was it a fungal infection? It was a disease.

Kip Boyle: It was a blight.

Jake Bernstein: It was a blight either way but it just tore through the-

Kip Boyle: Yeah, cut them off the knees.

Jake Bernstein: Cut them off the knees. There was no potatoes. And at that time in history, the Irish were relying almost entirely on the potato for their calories.

Kip Boyle: Yeah. And I could be wrong about this, but I think the Irish diaspora is enormous. I think there's more Irish living outside of Ireland right now than in it. And I heard it's a consequence of that.

Jake Bernstein: It would've been, yeah, there's no food. There was no food because all the potatoes died. And when they died, because they were monoculture, the whole concept of biodiversity is that all of your resources don't get impacted by one disease.

Kip Boyle: I'll tell you a quick story about this. This periodically comes up in conversations, and there was a time before Windows 2000 was released as a product. So this is 20 plus years ago.

Jake Bernstein: This is like anti era.

Kip Boyle: Yeah. So like DNS, yeah, Windows NT. So domain names system was all run on Unix predominantly. And I remember I was analyzing this new operating system to try to understand security implications and so forth. And one of the things that was new at the time was the idea that Windows would have these domain controllers that would actually control domain name system services, and that would push out Unix-based domain name servers. It would gobble up tremendous amounts of that infrastructure. And I remember Unix people howling at the thought that this was going to happen because they didn't trust Windows to be durable enough and robust enough to handle such an important service. And there was a lot of talk about monoculture that came up as a result of that. So anyway, here we are 20 plus years later, and that change actually happened.

Jake Bernstein: Well, I mean 100 plus years later. The potato famine, it's just so fascinating how you can draw... It's the same problem, Kip.

Kip Boyle: Yeah, conceptually.

Jake Bernstein: Conceptually it's the same problem. Yeah. I think those are, and of course, we had the obligatory congressional testimony. We had the executives come and plead with Congress people. There were huge impacts. We didn't really talk about it, but there were huge impacts to government around the world.

Kip Boyle: I think the Wikipedia page does a stunningly-

Jake Bernstein: It does a stunningly good job of-

Kip Boyle: Documenting that.

Jake Bernstein: Covering it all. Ground transport, healthcare, as you said, many hospitals had to pause non-urgent surgeries and visits. There were media and communications, stations were literally unable to broadcast, TV. ESPN couldn't air its morning edition of Sports Center, Kip. It couldn't.

Kip Boyle: Talk about critical infrastructure.

Jake Bernstein: I know. And it goes on and on and on and on. I mean, it was almost a worldwide vacation of sorts for a lot of people except the IT folks.

Kip Boyle: Yeah. Not at all.

Jake Bernstein: There's a lot to unpack here. In terms of the scale, I don't think anyone knows. There's nobody who doesn't know someone who is impacted by that. It was everywhere. It was completely, completely everywhere.

Kip Boyle: And I think a poster child, not the only one, but I think it's the most recent poster child for the NIST Cybersecurity Framework, which is all about cyber resilience for exactly these reasons.

Jake Bernstein: Yeah. We've actually talked a lot about what's in the script, but I want to scroll down to maybe two thirds of the way through for your information to where I've posted... Unfortunately, this is one of those situations where I wish we did have video to share with our listeners.

Kip Boyle: Yeah. You've got a good graph.

Jake Bernstein: But it's very easy. You just type in CrowdStrike stock into your favorite search engine, and you'll be able to find for yourself the same graph. I just put the year to date. And you've got CrowdStrike, let's see, it started the year at around 240. That was the stock market in general, just being low. It crawled its way up, let's see, into maybe a high of $340 in March.

Kip Boyle: And share gone up.

Jake Bernstein: By June there was, and really early July, there was a larger stock market rally. It peaked at about 300. What is that? 80 ish, and then, Kip, it is the roller coaster from hell. I mean, that is a slope of like, it's almost vertical.

Kip Boyle: It is. I was about to say it's as straight down as I've ever seen.

Jake Bernstein: Yeah, it's way down. CrowdStrike's stock drops from about 380 to less than 220 in about a week. That's a pretty massive cratering. However, it's back up to about 320. So yes, it is still lost about $60 a share. A real stock analyst would also be comparing this to the overall stock market. But I think it's fair to say that it wasn't the end of CrowdStrike, far from it.

Kip Boyle: No. But people were worried.

Jake Bernstein: They were worried.

Kip Boyle: So there are people who lost money because they did... The reason why the price went down is because people panic sold.

Jake Bernstein: They did. And I also want to give credit to where credit is due is, I think CrowdStrike's response, their contrition, the apology tour of the CEO, the congressional testimony, this wasn't just because time passed and people got over it. CrowdStrike, they did a lot to try to recover this.

Kip Boyle: I hold up one company in particular, Norsk Hydro as being an exemplar of how you handle public relations in the midst of a crisis. They had a ransomware event that shut them down. I mean, this is a multibillion dollar aluminum. They smelt aluminum and press ingots. And I mean, this is what they do. And they went completely offline. But the thing about them is that they handled it so brilliantly that when they were back, when they had all their services restored, their stock price was higher than it had ever been. So I think they were very good about the way they did it. And I think CrowdStrike in the same vein, has really done great. They leaned into it, and they took accountability for it. And they have provided assurance that they will learn from their errors.

Jake Bernstein: Yeah, agreed. So let's end by talking about this article that I found on TechRadar. The headline is that CrowdStrike Outage is Causing Businesses to Switch Security Vendors. Now, What's interesting here is that this isn't actually saying that the CrowdStrike outage is causing people to leave CrowdStrike. If you read the article, what it's really saying is that the event was a catalyst for causing people to reevaluate their security vendors overall. And it's almost 1 in 10, 10% of the organizations conducted by this German federal office. The federal office for information security was the research, or was the org that did the research. They said that 1 in 10 orgs affected by the CrowdStrike outage were dropping their current security vendor. They were switching.

And the really interesting part was that one in five of the companies that were planning to revise that did this, were also planning to revise their vendor selection criteria. So what does this say, Kip? This is good because what it says is that this incident caused people to look again, more closely at their processes and their procedures for vetting vendors. And again, this article isn't saying that 10% of CrowdStrike's customers are leaving CrowdStrike. Actually, it's more general than that. And what that means to me is that people are taking, at least for the short term, a lot of people are taking the right lessons. One could argue that 1 in 10 is not enough. It should have been 9 in 10 who are reevaluating.

Kip Boyle: Yeah. Well, now you're putting your finger on something that I don't think we've said so far, which is supply chain.

Jake Bernstein: Supply chain, supply chain.

Kip Boyle: Supply chain security, supply chain risk management, but also third party risk management in general. And that's why I said, in a way, if you squint and look funny at it could be considered a cyber attack. Because if you're Delta Air Lines, you could easily imagine that an outage like that they suffered could have been caused by a cyber attack. So I'm glad to hear not everybody is trivializing it by saying, "Well, it wasn't a real cyber attack, so it wasn't that big of a deal." Well, a real cyber attack could have been just as devastating and even more difficult because CrowdStrike cooperated in the recovery. A real cyber attacker is not going to.

Jake Bernstein: Exactly. Yes.

Kip Boyle: Yeah. And this reminds me of Kaseya. When they got exploited in 2021, a bunch of managed service providers became victims of ransomware because they used the MSP's to deliver that ransomware. We got the SolarWinds hack that happened in 2019, when they got exploited, and a bunch of malicious code was distributed through that software platform. And guess what? Kaseya, SolarWinds, those are all privileged IT management tools that I would put in the same bucket as CrowdStrike Falcon because we're doing infrastructure management and infrastructure security and that sort of thing. So CrowdStrike isn't even the first kid on the block.

Jake Bernstein: Not even close. No. And one thing I just realized I wanted to hit on as we wrap up here is that, there was also an interesting dichotomy between customers who had direct control over their own IT infrastructure. Even if it's cloud, the way it was put in some of these articles was that some customers who were directly using Microsoft, in other words, they were directly in charge of their own Azure environments or things like that, were actually able to get back up and running in literally minutes. So for them, it was no big deal.

Kip Boyle: And of course, anybody who wasn't using Falcon, and then I saw in the Wikipedia article too, and this makes sense. All the sanctioned countries in the world that couldn't legally purchase Falcon, they weren't affected.

Jake Bernstein: No. But the group that was hardest hit where it wasn't necessarily direct Falcon customers, it was those organizations that were fully reliant on a third party for their IT infrastructure. These are companies who have fully outsourced their IT to a managed service provider. Because think about it, Kip, if you're an MSP and you have X number of employees and you have X times 50 customers, and every one of those customers has gone down because of this CrowdStrike outage, and you have to put hands-on keyboard to resurrect the infrastructure, there were people who were down for days or even a week because they did not have the ability to fix their own problems, and it was just too much.

I talk about this with clients quite a bit. It can be "efficient" to outsource, but managed service providers as a class of business, historically, they've been smaller local companies, and I'm not knocking smaller or local managed service fighters. What I am saying is be careful and understand your needs and your IT needs. If you're using an MSP as a break fix helper, that might be okay. But if you've outsourced your entire infrastructure and manic control to an MSP, could be a real problem for you if you're just one of 100 customers when excrement hits the rotary air impeller.

Kip Boyle: Yeah, definitely. And even if you get guarantees from a managed service provider or an outsourcer about, well, if we have a failure, you will get some compensation. It's typically just fees paid, and we're not paying very much anyway, which is nice, but it's not going-

Jake Bernstein: Well, that's the point is you're not paying all that much. The other thing I say is you and I have always talked about security theater. This is where I warn people to think about what I call indemnity theater. That small local MSP, if their entire customer base goes down, they can't afford to pay everybody back for the failure. They just can't. It's just not going to happen.

Kip Boyle: That's where the insurance for you is a customer of the MSP come into play. Lots of revenue and so forth. Anyway, I know we could turn this into a five-part podcast episode if we wanted to.

Jake Bernstein: Yeah. We don't need to. Let's go ahead and wrap it up.

Kip Boyle: Okay. All right. Well, that wraps up this episode of the Cyber Risk Management Podcast. Today, we talked about the great CrowdStrike IT outage of July, 2024. I'm not going to put that as a title. Do I have to do that? Anyway, we talked about that. We talked about the impacts, the responses, and the lessons that we should learn from all of this. Thanks for being here, everybody, and we'll see you next time.

Jake Bernstein: See you next time.

Speaker 1: Thanks for joining us today on the Cyber Risk Management Podcast. If you need to overcome a cyber security hurdle that's keeping you from growing your business profitably, then please visit us at cr-map.com. Thanks for tuning in. See you next time.