Kip Boyle: Hi, I'm excited to share that we have a new free bonus just for you.

Jake Bernstein: We've started publishing our very own quarterly Cyber Risk Management Journal. It's loaded with over 30 pages of useful content taken directly from our podcast. Here's how we make each new edition.

Kip Boyle: We start by transcribing the four or five episodes that we've published in the previous three months.

Jake Bernstein: Next, we send our editor and designer the transcripts and our supporting materials for those episodes.

Kip Boyle: Then, they revise all the text. They put clickable links in for all the resources. They create the best look and feel for each episode.

Jake Bernstein: Finally, we, Kip and I make sure the finished PDF is ready for you.

Kip Boyle: Download the current edition now. All you have to do is go to b.link/CRMJ. That's the letter B.-L-I-N-K/C-R-M-J.

Jake Bernstein: If you like it, share it with your friends and encourage them to subscribe to the Cyber Risk Management Podcast. Now, on with the show.

Voiceover: Welcome to the Cyber Risk Management Podcast. Our mission is to help you thrive as a cyber risk manager. On today's episode, your virtual chief information security officer is Kip Boyle and your virtual cybersecurity council is Jake Bernstein. Visit them at cyberriskopportunities.com and focallaw.com.

Kip Boyle: Jake, what are we going to talk about today?

Jake Bernstein: Hey, Kip. Today, we're going to discuss one of my recent ransomware response cases and conduct a postmortem to see what we can learn in order to help our listeners prevent their own ransomware disasters.

Kip Boyle: This is going to be good. I think our audience is really going to appreciate this. If you are in the audience without having gone through a ransomware attack, this is going to be so helpful to you in terms of just preparing yourself, hopefully, to avoid it and certainly to prepare for one. I'm excited.

Jake Bernstein: Me, too. I'm already going off script here is that there's no substitute for this type of experience, but here's the thing, you really don't want to be a victim. The best way to avoid that, but still benefit from experience is to listen to an episode like this.

Where we're going to put you in the shoes of the victim company, who actually, we will refer to interchangeably as the client because they were my client or the victim. No names, of course.

Kip Boyle: I know from experience that a lot of people are not as interested in preventative activities. A lot of senior decision makers because they could do prevention activities on a whole spectrum of risks. What I've noticed is they tend to just assume that some things are going to happen and if they do well, we'll just clean them up.

That probably works for a lot of things. I think it's just a truism for entrepreneurs is that they can't fix everything so they have to pick and choose what they're going to fix or what they're going to prevent. I would say that of all the computer disasters that can befall you, this has got to be one of the worst. One of the ones you don't want to wait to fix until crosstalk

Jake Bernstein: We'll talk about it, but it's also one of the ones that in certain respects is also the most preventable. Maybe I don't want to say preventable because that is a little strong, but you can certainly prepare for ransomware in a way that you can't necessarily prepare for other types of computer-based attacks.

Kip Boyle: You can definitely reduce the risk. You can.

Jake Bernstein: Here's a quick rundown to set the stage. Kip, you'll go ahead and give the audience our focus points and we'll, then we'll go through and discuss that.

Kip Boyle: Sounds good. Take it away.

Jake Bernstein: Here's the main timeline. Late October ransomware hit and encrypted all servers of the victim and completely shut them down. Within a few days outside counsel, not me, had been retained and an incident response company had been hired. Almost within a day of that outside their primary council then called me in to assist.

Now, here's an interesting thing within two or three days of hiring that incident response company, there were sufficient problems that the victim made, in my opinion, drastic decision to hire a second incident response firm. The client had the idea that both incident response firms would be working on the case, but that didn't last more than a few days.

It was pretty clear too many cooks in the kitchen. Once the second incident response firm was settled into place, a multi-week process of forensic investigation and recovery began. All told, the victim was basically completely out of commission, out of business for at least three weeks, although they did start to get at least email and basic functions back.

That's the summary. Now, obviously, there's a whole lot of detail. We will talk about it. Why don't we go ahead and go through our list of what we wanted to focus on.

Kip Boyle: Even after you were just going through that very abbreviated timeline, I had to sit here with my hands. I was sitting on my hands going, I want to ask about this. I want to ask about that. If we did it that way, this episode would be very, very long.

Rather than do that, what we're going to do is we're just going to five things that we think are most relevant and we're going to focus on those. The first thing that I think we ought to talk about that matters the most in terms of helping our audience prepare for this kind of an experience is the false start with the first incident response company.

That really strikes me as being a major mistake. Probably, on par with the original mistake of somehow getting infected with ransomware. Tell us more about that.

Jake Bernstein: The fall start was fundamentally, the victim didn't have an incident response plan. Right away, that's the big issue there. You can think of it this way, incident response companies they're like fire departments except you have to pick your fire department. It's not as simple.

It's not considered a service, at least not yet. I think we talked about that at one point. You really shouldn't go shopping on Google for an incident response company.

Kip Boyle: At the moment you need them.

Jake Bernstein: It would be like shopping for that fire department while the house is literally burning. Not good timing. That's the basic easy part is have an incident response plan and figure out who you're going to use ahead of time, whether it's with outside counsel or not.

Kip Boyle: Your insurance company. A lot of insurance crosstalk policies is going to have this as an essential service built into the policy.

Jake Bernstein: Well, spoiler alert here, the victim didn't have cyber insurance.

Kip Boyle: Another massive misstep on their part. I feel bad for them they probably didn't even realize.

Jake Bernstein: Probably not. Now, all of that being said, that's not why we had a false start with this company. I think the issue really was performance. Obviously, I'm not going to name names here, but what I do want to point-

Kip Boyle: Wishes you would, but I know crosstalk

Jake Bernstein: I know, but I can't. What I can do is point out the critical difference between the two IR companies. That is what I've recently learned is the difference between a tools-based approach and a skills-based approach to incident response. This is I think really important.

A tools-based approach requires personnel who know how to operate a certain piece of software. That's the tool that I'm talking about. Usually, it's something like a Carbon Black or a Microsoft ATP, which just stands for Advanced Threat Protection, but basically this whole EDR new form of EDR, which I think stands for Endpoint Detection and Response software.

That's the most common tool that I'm talking about here, but when an incident response company's primary strategy is to install an EDR software and let it do its thing you're really missing out on a lot and it's a universal problem. Compare of the skills-based approach where you have frankly, trained hackers and anti-hackers, or I should say counter hackers, like sniper and counter sniper.

Kip Boyle: Intelligence, counter intelligence.

Jake Bernstein: Exactly. There's no substitute for just raw experience. Tools are great don't get me wrong. Just to absolutely clarify I'm not criticizing Carbon Black or Microsoft ATP. Heck, both of those were used in this particular case, we ultimately used the Microsoft ATP software, because it came with the company's Microsoft 365 subscription.

The software is not the problem. The software was great. It did its job. The problem is when you rely upon that and don't know how to do your own independent investigation. The difference was stark. I saw the initial company, we had status calls and more or less it was well we installed this piece of software.

We're letting it do its monitoring thing and trying to cleanse. I don't think that's 100% fair. I think that this incident response company did a little bit more than that. By comparison to the second one that ultimately was retained for the long haul, it was like night and day.

The second company had experts who had their own scripts and built their own scripts just for this client. They knew what they were doing. They knew how to look for things. It was a lot of work. That's the thing that really struck me the most about both the forensic investigation and the recovery side is it is a lot of work.

We're talking hundreds and hundreds and hundreds of man hours of time between digging through log files, reconstructing complete network infrastructure. Rebuilding virtual machines. It is a ton of effort and work. One of the things that you want to think about is how big it is my network? How many people do I think it's going to take?

Then, when you hire an incident response firm, one, ask about not just the forensic side, because that was another issue here, which I'll get to in a moment. Also, the recovery side and how many bodies the incident response firm can throw at the problem.

Kip Boyle: Let's recap before we keep going. Let's recap.

Jake Bernstein: Let me say one more last thing on this because it is important is that part of the problem with the first incident response company and this wasn't really their fault, although it depends crosstalk.

Kip Boyle: It's ultimately on them.

Jake Bernstein: It was. They overpromise, but I also think the client misunderstood. The problem was this is that the client thought they could hire one company to do the forensic investigation and the IT recovery. While the first IR firm said that it could handle both, what it really meant was, "We can do the forensic investigation. We've got some contractors that we work with who can do the recovery."

The problem is that during that time period, there were so many ransomware attacks. Particularly, on hospitals and elsewhere. This would've been in late 2020 for those listening in the future. There was no one available and that became a major problem. Now, we can go ahead.

I just want to point out that when you're interviewing incident response firms ask about both forensics and recovery.

Kip Boyle: Wow. In addition to all the other issues, there was just a flat-out excessive market demand for the supply of people who could do that work. Is that what you were saying in this last point?

Jake Bernstein: Correct.

Kip Boyle: One of the things you were saying. That's really interesting because I continue to say that the cybercrime wave that we're living through right now is wildly under reported in the news media. I'm on guard to find out like, "Well, why is it under reported?" Also, look for evidence that it really is substantial.

A thing that's happening to us despite the fact that it's not being reported very well or completely. That's just another example, you wouldn't necessarily know a senior cyber risk decision maker wouldn't know that there's a lack of vendors until they tried to go get one. They probably wouldn't know. I don't know.

Jake Bernstein: There'd be no way to know. Honestly, the incident response from itself didn't even know. It was happening literally by the hour.

Kip Boyle: Wow, that's amazing. Let's recap the first thing. There was this false start with the first incident response company. I think it's totally reasonable to say that there were problems on both sides. The buyer not really understanding what they were trying to buy. The seller, not really selling something that was going to do the job.

Then, bringing in the second company. You were talking about a tools-based approach. I don't know that I heard you say it. I just want to be totally clear. You're saying that the first IR company was saying, "We're going to use tools-intensive approach to get you out of this ransomware situation."

Jake Bernstein: They didn't say that. I don't think anybody says that. That's a good point of clarification. I think when you're interviewing, you should ask about the training and the experience level of the actual responders and try to discern, "Is this company selling me a tools-based approach or a skill-based approach?"

You may not be able to get a straight answer, but if you're the customer it's definitely worth asking. This is another reason, if you have the time, when you're doing an incident response plan, then it should be no problem to figure out find a good company ahead of time.

Kip Boyle: I think that's one of the big takeaways is don't fumble around looking for an incident response in recovery firm when you need one in the moment have that sorted out in advance. I really think the first probably best option is just that a really great cyber liability policy.

The insurance company will have already vetted all of this for you. They're only going to send people who are qualified. That's what I think.

Jake Bernstein: I'm not sure that I would agree with that. I would hope that's the case. I will say this, the incident response firm that was hired was a big name in some cases. I think in the right circles it's a big well-respected name. The problem is that this is a broader cybersecurity problem is that it's such a tempting market space that so many companies are trying to generate incident response capability.

There's not enough people with skills. You have to default to a tools-based approach. That's the broader issue. I would just caution you even further crosstalk

Kip Boyle: Immaturity. There's a lot of immaturity.

Jake Bernstein: It is. Definitely, the insurance company will help, but it's on you to find your preferred provider who you know is going to do the best job. If you're okay with a tools-based approach, then it doesn't matter as much, but that isn't necessarily the best possible incident response plan.

Kip Boyle: That's the first thing. Now, the second thing that we want to point out from the timeline and from what you saw, and by the way, I didn't work on this case. I learned about it after the fact. I also learned about it through confidentiality filters too. I wasn't working on it. I had no right to know least privilege and all that.

The second thing was the ease of infection. Jake, tell us how the ransomware got started.

Jake Bernstein: This is a really interesting and I think scary set of facts. There was basically a laptop used by a typical, I think it was a mid-level manager. Eventually obviously, this is all stuff that we learned after the fact, hence why this is a postmortem. The patient zero, which was this laptop was infected at approximately 3:38 and 31 seconds.

Kip Boyle: Approximately.

Jake Bernstein: Approximately on a date in late September. What this means is that's when the malware was downloaded. Within nine seconds, the malware had been executed on patient zero. Within five minutes, a second stage of payload in this case, Cobalt Strike was launched on patient zero.

Then, within seven minutes more, the malware began initial network discovery. A few minutes after that, literally about five, there was local escalation and credential dumping on patient zero. This took a bit longer, about five hours later, we were able to confirm escalation to a domain admin account.

Over the next few days, there was additional discovery RD ping to servers, installing and executing Cobalt Strike on servers. There was a bit of a lull of about five days where we believe that the bad guys were just distracted by something else, a different victim.

It was about a 20-day span of internal data discovery activities, port scanning, web browsing, all that good stuff. Uses of Mimi CADs, which is a common tool to dump passwords from memory. At the end of that 20-day period, this was late October the key day was when they did a global webroot uninstallation that's to clear the way.

I think webroot is an AV software. That cleared the way for the actual one, they deleted at cloud resources. It was a big one. If you think you have the cloud you may not. Then, and only then did they do the ransomware encryption. Just to summarize, the entire process was less than a month, but the critical time period happened in minutes to a few hours.

Kip Boyle: That's when the initial malware gained its foothold and set itself up to be able to do all the other things that it needed to do to prepare for the attack, to launch the attack.

Jake Bernstein: It's worth pointing out too, the way it started was the client had a customer whose website was compromised. While the client was browsing its customer's website the person received a popup message saying, "Your Chrome browser is out of date. It needs an update."

"Gosh, updating is important. I'm going to do that." It turns out obviously it was a fake download. It was a zip file with JavaScript inside and that started the whole thing.

Kip Boyle: Let's see. The ease of infection, it really starts with this person driving this laptop who browses. This person's conducted in a legitimate business activity, trying to surf the website of their customer. Then they get this prompt and to do an update, and then they think, "That's good cyber hygiene. I should update my software. I should stay current."

Then it just rolls from there. It sounds like it was essentially silent. Everything that happened in those first 20 days happened without notice by anybody. Is that right?

Jake Bernstein: Yes, that is correct.

Kip Boyle: Did you guys ever talk about when it could have been noticed? What were the missed opportunities to notice it? Did that ever come up?

Jake Bernstein: It didn't really come up in the context of the incident response primarily because probably, it didn't really matter at that point. Definitely, in terms of what to do about it going forward there was discussion on that. My impression is that the victim didn't have the cybersecurity capabilities that were even remotely necessary.

There's not much to learn on this because they didn't have much to begin with. Had they been running an EDR piece of software, then they would've potentially been caught.

Kip Boyle: Well, they had webroot.

Jake Bernstein: They did have webroot, but what you'll hear about more and more is because of the way so much of these ransomware, and then by the way, this was a sophisticated actor. This was not malware as a service or ransomware as a service. We'll never know exactly who, but it was a well-known and sophisticated group that is the likely cause of it.

Kip Boyle: There was a crew of experienced operators making this happen. That really underscores a point that I try to make a lot, which is we're dealing with people here. It's easy to think, "This is just a computer problem. This ransomware is causing all the trouble." Behind all of that are people.

They're just going to keep innovating and finding new ways to attack us and overcoming and any obstacle in their path as they do what they do. We certainly saw that here where you did. We've got five major points. The first one was the false start with the IR company. The second one we just talked about, which was the ease of infection and the lack of the detection.

The third one, which I thought was fascinating is the Laundry Network that was created for the recovery process. I hadn't heard that term before. What's the Laundry Network, Jake?

Jake Bernstein: The Laundry Network, I think other people might call it a quarantine network or something along those lines, but what it is a method to use when you're bringing servers back online. This was largely because the backups had been deleted. The cloud had been deleted.

Though I might as well just mention it now, the victim had tape backups, but they stopped updating them two years ago. They didn't really have tape backups.

Kip Boyle: It was so stale as to be unusable.

Jake Bernstein: Obviously, the takeaway there is you've got to follow the good old 3-2-1 rule with your backup strategy. All that really means is three copies of all your data. One of which is production, on two different medium or media, and then one should be offline.

This is why you want a copy offline because if something is online, it is inherently vulnerable to attack.

Kip Boyle: The first versions of ransomware didn't really do that. If you had a backup, it was relatively easy to dodge ping the ransom, but then the ransomware got really good at detecting and destroying backups that were online.

Jake Bernstein: Well, I mean, someone wise wrote a book with the title Fire Doesn't Innovate and the corollary is that hackers do innovate all the time, constantly, nonstop. Ransomware is a good example of that.

Kip Boyle: It's really forced us back into old things that we thought we'd never do again like tapes.

Jake Bernstein: Well, but the thing about a tape is that it's offline and you can't do anything to it once it's made. It might be a pain, but man, is it effective as a backup strategy.

Kip Boyle: I remember the days of tape and we were all happy to leave tape behind us at one point, but we didn't realize we were walking into a trap. Welcome back, tape.

Jake Bernstein: We say tape, but there's different ways. You don't necessarily have to use old school tape. There's other ways that you could set it up. The bottom line is that one has to be offline.

Kip Boyle: It's still a pain in the ass.

Jake Bernstein: I know it is, which brings me back to the Laundry Network. The point of the Laundry Network is to be able to bring servers back online that have been cleaned. Not necessarily fully wiped, but cleaned using an EDR software of some kind. When I say cleaned I don't know the exact technical implementation going on here.

My impression though, my best understanding is that some of them were re-imaged. What you had to do was the Laundry Network was a very, very limited pipe out to the internet. Literally, all it could do was access Windows update. What you would do is you would allow all of these machines to get updated.

Part of the reason, by the way for that was that when they weren't re-imaged because I think the reimaging was considered pretty safe. If you re-image a machine it's new. For the ones they couldn't do that on, one, they had to get access to the windows update and EDR service because that's how those things work.

Then you have to let them percolate or I suppose you could say launder and then dry, which just means give them time. What you're doing is looking for attempted outbound communications to the command and control server. You do your best to clean it off and then you let it sit and watch it.

You do that in the Laundry Network where you can't do any harm and that's the idea. Then once something has passed through the laundry network, which it usually is a day or two at minimum, then you go ahead and you can start to move it toward production.

That's what a Laundry Network is. It's a lot of work, it's a ton of work to set this all up technically. Conceptually, it's fairly simple, but it is not simple to execute. I think different companies will call it things, but that's really what it is. It's a methodology to clean servers.

Kip Boyle: It's so concrete. When you first mentioned the Laundry Network, I was like, I just had this image of industrial giant washing machines that you would throw banquet linen into or something like that.

Jake Bernstein: I know, it's funny. I think an interesting point here is this entire incident response was conducted remotely. I didn't mention that at first, but I think it's worth pointing out. First of all, COVID definitely played a role in that, but it's also nice to know that it's possible.

Now, that did cause some problems here and there. I mean not having access to the network infrastructure can be challenging. Definitely caused some slowdowns, but it still worked.

Kip Boyle: I don't think it betrays any confidences to say that the IR come was on a different continent than the victim.

Jake Bernstein: They were.

Kip Boyle: Just to say that, just so people understand that when you say they didn't have physical access, they really didn't have physical access.

Jake Bernstein: They really didn't have physical access. That is correct.

Kip Boyle: It's global IR, which is interesting. That's the Laundry Network, that was the third thing. The fourth thing we've already talked about, but is that their data backup strategy failed. Is there anything else to say about that?

Jake Bernstein: No. It just didn't work. The ransomware found all the backups and killed them before it started the final encryption process.

Kip Boyle: I will add the idea that if you are going to make data backups you do need to test them. You do need to test them to make sure that they work. There's been lots of stories of people who made backups, but especially in tape land, and then found out when they tried to do a restoration that the tapes were either empty or formatted in such a way that didn't allow the data to be restored.

The data that was backed up wasn't quite right for whatever reason. Data backups can fail or they just aren't created correctly and so you really do need to test them. Then the fifth thing is the role of the attorney. Listeners might be thinking, "Well, this is really interesting." The attorney in the podcast is the one that participated in this, not Kip. Why, Jake?

Jake Bernstein: A lot of reasons, really. To begin with, it's attorney-client privilege, as you know, as we've talked about. The lawyer needs to initiate contact and liaise with law enforcement. Help with internal and external communication plans. A big part is to determine reporting obligations under local state, federal, and international law.

That is not as simple as it sounds. There's all of pieces of analysis that goes into that question. I think a really interesting piece is to help the client navigate cryptocurrency and the risk of sanctions under the OFAC list. That came up in this case. One of the things that you wanted to talk about was payment.

Early on the victim was assuming that it had to pay. The question was, "How do you pay? That's when we ran straight into the brick wall of the October 1, 2020 OFAC or Department of Treasury memo about the OFAC list, which just means basically you can be sanctioned for paying ransoms to any person or organization listed on the OFAC list.

In some ways adding insult to injury, you're already hurting. Now, the government's saying, "By the way, "We don't really want you to pay the ransom and get your stuff back assuming that it worked." The way around it is to work with law enforcement. It's not a hard and fast rule.

I think that's a major component of counseling between with the lawyer. Then insurance coverage issues broadly and analyzing third party liability. That's a big part as well. What can go wrong? What if your infection leads to another infection, et cetera.

Kip Boyle: There's a lot there and I wanted to clarify a couple things. OFAC is the Office of Foreign Assets Control that's what that little OFAC acronym means. It is part of the Department of Treasury. I would recommend that if you haven't seen that memo that came out, you should go grab it and study it.

The practicality of it is here's the U.S. government trying to sanction other countries for doing things that the U.S. doesn't want them to do. For example, North Korea, just to make a big example here, they are pursuing nuclear weapons and they've been sanctioned quite a bit.

They still want to pursue this so they're looking for hard currency, well, guess what? This is how they do it. This is one of the groups out there that is trying to raise hard currency. I don't think it's a surprise that the group that is monitoring economic sanctions would be interested in this since it's become such a revenue stream for these sanctioned countries.

Then the other thing I want to point out is we did an episode of this podcast a while back, it dropped on November 24th when we talked about that there were new rules for attorney-client privilege over data breach reports. We were looking at the Capital One case at that time.

Did you have to factor any of those new rules into the ACP that you were helping to provide in this case?

Jake Bernstein: We did. Yes and no, that definitely informed our entire approach in the bizarre twist, because the company didn't have an incident response plan and didn't have an IR company on any contract, the exact facts of the capital one case were impossible to happen. Otherwise, yes, it definitely, it informed a great deal of how we operated.

Kip Boyle: That's great. You were successful in established ACP as far as you know.

Jake Bernstein: As best as anyone could, yes.

Kip Boyle: In uncertain times in uncertain situations. Anybody who didn't listen to our episode from November 24th, New Rules for Attorney-Client Privilege to Data Breach Reports, go check that out. Those are the five top things that we wanted to really bring out from Jake's experience on this ransomware case. Any final thoughts, Jake?

Jake Bernstein: Just get a backup strategy in place. That's so critical.

Kip Boyle: A data backup strategy and have that IR plan. Test it, it's really not that tough, but it's one of these important, but not urgent things. You can really have a hard time getting it onto the calendar of your senior decision makers. I face this all the time. I totally understand that it can be difficult, but we're here to tell you that it's totally worth it.

Jake Bernstein: It is, very much so.

Kip Boyle: Please do that. Well, that wraps up this episode of the Cyber Risk Management Podcast. Today, we discussed a modern ransomware experience that Jake went through. We shared with you some really important postmortem lessons to be learned. Thanks for being here. We'll see you next time.

Jake Bernstein: See you next time.

Voiceover: Thanks for joining us today on the Cyber Risk Management Podcast. Remember that cyber risk management is a team sport, so include your senior decision makers, legal department, HR, and IT for full effectiveness. If you want to manage cyber as the dynamic business risk it has become, we can help.

Find out more by visiting us at cyberriskopportunitiesdotcom and focallaw.com. Thanks for tuning in. See you next time.