Speaker 1: Welcome to The Cyber Risk Management Podcast inaudible mission is to help you thrive as a cyber risk manager. On today's episode, your virtual chief information security officer is Kip Boyle, and your virtual cybersecurity council is Jake Bernstein. Visit them at cyberriskopportunities.com, and focallaw.com.

Jake Bernstein: So Kip, what are we going to talk about today?

Kip Boyle: Hey, Jake. Today, let's talk about how cyber attackers are exploiting the trusted relationship between IT service providers, and their customers, which are members of our audience, and what you should do about it.

Jake Bernstein: Okay. Well, I guess, let's start by asking the obvious question. What do you mean by IT service provider? That term could mean anything.

Kip Boyle: Yeah, actually it covers a lot of ground and I chose it deliberately because IT service provider is really three major buckets of types of companies. But an IT service provider, from our audience's perspective, is going to be an outside entity. So, again, there's three types. So, the first type is what we call a managed IT security services provider, right? So, MSSP. And a good example of that would be a Symantec. There are many examples, but Symantec is a good example. They provide what's called intrusion detection system management. So, if you want to be able to detect intruders on your data network, but you don't want to invest all the money in the equipment that you'll need, and then a team of people to operate that equipment, you can turn to Symantec, and they will provide that for you as a service. So MSSP, there are many different MSSPs out there.

The second type of an IT service provider would be a Cloud services provider. And it's funny, there are so many Cloud services providers out there today, and a lot of people don't even realize that they use a Cloud Service Provider, because it's just so woven into the fabric of our lives. But Gmail, for example, Office 365, those are Cloud Service Providers. And another example is if you buy, say virtual servers from Amazon Web Services, AWS. So, just a couple examples of Cloud Service Providers. And then the third type is what we might call a managed IT services provider. So, this is a very basic type. And an example would be, let's say you're a dentist, and you don't want to really monkey around with computers, you just want them to work. And so, you might hire an outside provider to do, let's say, medical records, retention, and backup management for you. And a good example of that is actually a company that was very badly exploited recently called DDS Safe. So, those are the three types of IT service providers, then there's thousands of them.

Jake Bernstein: So, are we including in the MSP category, the kind of small 10 to 40 person outfit that there's just hundreds of them in the Seattle area. They're probably 20 years old and they just provide, they're kind of outsourced IT, and they all call themselves MSBs. Are we including them in this conversation?

Kip Boyle: Yeah, definitely. And you're right. They're as numerous as mushrooms after rain, they're all over the place. So, yes, no matter how big, like Symantec's quite big, right? And then you've got IBM, and other very, very large companies. But, oh my gosh. Yes. There's so many smaller ones around as well.

Jake Bernstein: Got it. Okay. So, obviously, these are useful services...

Kip Boyle: Yeah, absolutely useful.

Jake Bernstein: People need them, right?

Kip Boyle: Yeah. They really do. I mean, just going back to that example that I gave about DDS Safe. I mean, if I'm a dentist, I want to be focusing on patients, right? I don't really want to become an expert at medical records, retention, and data backup, and all that stuff. Those are very complicated, and from a dentist point of view, esoteric skills. And a dentist doesn't really benefit by putting their time into learning those things. So, it's great that there's a service out there that they can hire for far less money than they would have to spend on their own. And so, it's a big part of our economy in the United States.

Jake Bernstein: Yeah. I mean, and this idea of outsourced, this is everywhere. You can get lot of back office, functionality, HR, things like that are all kind of coming from different managed service providers.

Kip Boyle: Right. Software as a service is a great example. And it's to the point now where when you build out a... Let's say, a branch office, you're a large company and you build out a branch office. A lot of companies aren't even installing servers anymore. I mean, it used to just be... It used to be a given that you'd have a data closet with a few servers in there, and you'd have a local IT person rotating backup tapes and all that. And that doesn't even happen anymore because everything that you would put in that data closet can now be purchased as a service from a Cloud provider.

Jake Bernstein: Yeah. Though it is not the topic of this episode, it's interesting to note that we're starting to see a little bit of a reversal of some of that just because turns out tapes have benefits. But-

Kip Boyle: Right?

Jake Bernstein: But let's go ahead and move ahead here. So, obviously, and I just hinted at this with my kind of non sequitur comment there, there is a dark side to this, isn't there?

Kip Boyle: Yeah, absolutely. And that's what we want to focus on on this episode is the downside. And so, let's talk about, let's unpack that a little bit about why there's a downside, and how that happens. So, it's probably not difficult to imagine that in order to be useful, an MSP has to establish a trusted relationship between themselves and their customers. A part of that relationship is contractual, right? So there's going to be a contract. And then part of that is technological.

Jake Bernstein: Yeah.

Kip Boyle: So, if we take a look at a company like DDS Safe, a dentist that says, "Great, I'm going to use this MSP to protect my records," the dentist has to allow the DDS Safe, full access to all their dental and business records. It's just, they can't do the work unless they get that. Now, typically, this is done be through a piece of software. So, DDS Safe will say to the dentist, "Here's a URL, go ahead and navigate to it, download software, put it on your computer, put it on all the computers that you want to do record retention, and data backup on. And then that's how we will reach into your computers and back up your data, and we'll set a schedule," and so forth. And so, what's happening is cyber attackers are compromising that trusted network connection between the MSP, and the dentists' computers.

Jake Bernstein: Yeah. And I think I wanted to point out too, that when we say establish a trusted relationship, we're using that in not just the typical English language sense, and a legal sense, but also the more technical security sense. And I don't necessarily even mean that as a like its ones and zeros technical, I mean that as a security concept. And the important idea here is that of a trusted third party that for example, the public key infrastructure uses this concept. And that's really what an MSP is needing to do. I mean, whenever you sign up with an MSP, you really are trusting them. And I think that that is something that should not go unsaid.

Kip Boyle: Right. Well, yes, it's a great point because if I'm the dentist and I'm hiring an MSP to manage my records, it's almost as if I had hired an employee, because I'm giving them access, and emotionally, I'm making a commitment to them as well as a financial commitment. So, you're really inviting this group of people, most of whom you've never met. Maybe you've met the account executive, perhaps you met like the lead technician, but most of the people who are going to be providing the service, you've never met, you wouldn't recognize them on the street. You have no idea exactly how they do any of their work. And so, there's a big leap of faith here.

Jake Bernstein: Yeah. There is. Okay. So, how long has this been going on, this pattern?

Kip Boyle: So, the exploitation of MSPs has been going on at least since 2009. So, we're talking about a decade. And the reason why 2009 is an important focal point is because that's when there was a group that was first identified and tracked by US law enforcement. And the group goes by the incredibly nerdy moniker of advanced, persistent, threat 10 group, or APT10. And yes, because there's a 10, you can an infer that there were nine others before them, and probably, a whole number of them afterwards. But APT10, turns out after a long time of tracking them, we figured out that it's really a team inside the Chinese ministry of state security. And in 2018, the US department of justice actually indicted two members of that team by name. So, we're talking here about the exploitation by a foreign government is one of the threat vectors here.

There's another group called the Lazarus Group, and they're actually at team of people in the North Korean army. So, you're getting China, you're getting North Korea, you're getting a lot of different state actors participating in this. And then there's also what you might call, well, what I might call, common organized cyber criminals, who maybe only have a loose affiliation with their governments. And a great example of this is a guy named Bogachev, who runs an entire team of cyber criminals, and he operates out of a mansion on the Black Sea. He's actually on the FBI's 10 most wanted list, and he's stolen hundreds of millions of dollars using malicious code.

Jake Bernstein: Yeah. Well, and if he's smart, he'll stay out of extradition countries.

Kip Boyle: Which a lot of his buddies have not done a very good job of doing.

Jake Bernstein: Yeah. Okay. So, we've got everything from the Chinese military, to the North Korean army, to Russian guys living around the Black Sea. How-

Kip Boyle: The Millennial Mobster.

Jake Bernstein: Yeah. Yes. How do they compromise an MSP?

Kip Boyle: Well, actually, there's a pretty typical attack pattern, and it typically starts with a phishing attack. And we've talked about this before. Something like 90% of all cyber attacks start with a phishing attack, and why? Well, because it works. And so, a phishing attack... And they can just sit there, and just keep sending phish emails until one of them works. And it's inevitable that one of them will work. When you talk to the companies out there that are selling like anti phishing training services, they'll tell you something like, "Well, we can take your click rate from about 20%, down to 2%, or 3%, or something like that," and that's a great improvement. But there's still 3% of all the phishing emails sent to that company are still going to be clicked on. So, that's why we continue to see it happening. So, it starts-

Jake Bernstein: It only takes one.

Kip Boyle: It only takes one, and I can send a million for almost no money. And while those million are being sent, I can be off doing something else, sailing on my yacht, attacking somebody, else who knows?

Jake Bernstein: Yep. Sending another million to someone else.

Kip Boyle: It's easy, especially because I stole a credit card, and I'm having a virtual server in AWS do it for me. So, no money.

Jake Bernstein: Yeah.

Kip Boyle: So, it starts with the phishing attack. And then, that's how the initial access to the MSP gets established. From there, what happens is more malicious code is silently installed to monitor the computers of the MSP to do remote monitoring, and to steal user credentials, and ideally, administrative credentials. Right? So, they get in, and then they stock the MSP. And then after they succeed in stealing administrator credentials, then they will use those credentials to compromise the MSP systems, and the networks of their customers. And again, they'll continue to monitor, right? This is not the kind of malicious attack where they're holding your computers for ransom necessarily. Again, it just depends on what they're up to, but they're not going to make their move. Either steal data, or launch malicious code, to encrypt everything.

They're not going to do that until they fully perform reconnaissance, right? So, this is very much a military style of attack. And after they identify the data that they want on the compromised computers, they package it up for exfiltration. And exfiltration is just a fancy word for saying, "I'm going to get it out of their network." They'll typically encrypt the information that they find, and they have all kinds of different ways of exfiltrating the data, but they'll use their stolen credentials. And they'll actually hijack existing systems to move the data out. And that's called living on the land. So, if you ever hear somebody in cybersecurity say that an attacker is living on the land, all it means is that after they steal administrative credentials, they can actually use existing systems to get their work done. And it helps them blend into the overall noise of the network because they're not using special tools that would stand out if that makes sense, and then they're gone. So, that's the typical attack.

Jake Bernstein: Okay. So to what end? What do these cyber attackers want? What are they getting?

Kip Boyle: Yeah. So, it depends on who's attacking you, and unfortunately, you don't get to choose. So, if we talk about APT10, which is the group from the Chinese military, their major goal is they want to give Chinese companies a greater economic advantage in the world market. And so, they want intellectual property, and they will target specific industries, and specific companies in a order to do that. There's a great example of US steel making companies being targeted, and having the formulas, and the know-how for steel making stolen, and then provided to Chinese steel makers, and then having them flood the world market with cheaply made, yet high quality Chinese steel. So, that's one motivation.

I mentioned the Lazarus Group, the North Korean cyber attackers. And they're not really looking for economic advantage, they're actually stealing hard currency. Well, they're stealing Bitcoin and collecting ransoms in Bitcoin because they want to convert that into hard currency. With all the sanctions that are on North Korea they're really struggling to find a way to fund their nuclear weapons program. And so, that's their major motivation. And cyber mobsters like Bogachev, I guess, what? Yachts and pinky rings? I don't know. I mean, can you ever have enough of those?

Jake Bernstein: Drugs, houses, everything. Yeah.

Kip Boyle: Everything. Yeah. Mansions. I mean, they have an appetite. I'm reminded of a quote from the matrix where the Oracle says, "What do all men with power want? More power."

Jake Bernstein: Yeah. Yeah.

Kip Boyle: So, I guess it's about power. Anyway, so yeah, those are the common motivations.

Jake Bernstein: Okay. So, what should the MSPs do to protect themselves?

Kip Boyle: Right. So, and this is where we get into some shaky ground, because everybody has an opinion about what you should do. And I'm definitely going to tell you my opinion here in a moment, but first, I'm going to point out that The Department Of Homeland Security has issued alerts about this. And the most recent one I looked at is called, and get ready for this, TA18-276B. So-

Jake Bernstein: That sounds very official.

Kip Boyle: Great marketing on behalf of the Department Of Homeland Security, so that's Tango Alpha 18276 Bravo, in case anybody wants to look that up. And it contains several pages of very specific recommendations. Some of them very, very technical in nature, some of them more procedural. And I don't have a problem necessarily with what's in there, they talk network architecture, virtual private network connections, event logging, and so forth. But an MSP could spend... Or a customer of our audience who's trying to defend against this could spend millions of dollars implementing that stuff. But my opinion is that since these attacks all commonly start with phishing, or by exploiting missing patches for known vulnerabilities, I would think that an MSP would be better served by starting to secure themselves using the Essential Eight, which are cyber hygiene practices that have been published by the Australian Signals Directorate.

And the ASD is kind of like the Australian version of the American NSA, so highly technical kind of a spy organization. And they've published this list of Essential Eight things. And the Essential Eight is designed specifically to reduce, greatly reduce the risk of malicious code infection. And that's really what we're talking about here. So, that just seems like that's a more economical starting point.

Jake Bernstein: Why don't you go ahead and give me a few examples of these Essential Eight practices?

Kip Boyle: Yeah, sure. So, in priority order, that's the other thing I love about the Essential Eight is they'll actually give you a priority, like do this first, do the second, do this third. It's very prescriptive, but also very effective. So, in terms of malicious code protection, the first thing you're going to want to do is data backups with offline protection. And offline protection is a specific guard against ransomware, and modern ransomware is going to find your backups, and they're going to encrypt them.

Jake Bernstein: Yeah. In fact, that's one of the reasons why tapes are making a comeback in a sense.

Kip Boyle: That's right. Yep. Because, I mean, it's very difficult, maybe impossible, for a piece of malicious code to hop out of computer, run across the floor, and infect your tapes. And so-

Jake Bernstein: Yeah, that's the idea.

Kip Boyle: Yep. So, now, offline protection is going to introduce a certain amount of clunkiness, and is going to increase your costs because you're going to need manual intervention, or you're going to have to go find some kind of a tape jukebox to kind of automate the use of tape, but tape's slow. Very, very slow. But yeah, I mean, sometimes, the old ways are the good ways.

Jake Bernstein: Yep.

Kip Boyle: So, that's one high priority practice. Another practice out of the Essential Eight is called application whitelisting. And I love application whitelisting because in my perspective is that blacklist, which is kind of what antivirus products are, is they maintain this enormous list of things that should not run on your computer because they're known to be malicious. Well, I mean, that list is getting longer, and longer, and longer, and longer, and longer. And then the other problem is that malicious code is being turned out now in a way that every individual instance of the same virus, has a different signature, has a different fingerprint. And there's just no way for a black listing approach to be able to deal with all that. And that's why we're seeing the introduction of so-called artificial intelligence, or behavioral analysis, as a way of augmenting the blacklist approach.

But I love whitelisting because it's so much more straightforward. You just figure out what you do want to run on the computer, you can figure your machine to enforce that, and off you go. It's not foolproof, I don't want to pretend that it is, but I think application whitelisting is something people should be looking at. We implement this for our customers, and on the surface, it seems daunting, but the truth is that it's not as bad as you think. The reality is probably somewhere in the middle. And then-

Jake Bernstein: And it pays huge dividends-

Kip Boyle: Huge.

Jake Bernstein: ... in terms of preventing the execution of many types of malicious code. Obviously, if you allow Word to run, and there's a macro malware, then it may still run, but a lot of other stuff cannot.

Kip Boyle: Right. And there's also a benefit to the technical team too, which is by implementing application whitelisting, you're going to actually stop people from downloading installing stuff that really just makes your life more difficult as an IT professional. If you don't want them installing games, if you don't want them installing other web browsers, pirated software, you name it, application whitelisting's going to put a stop to that. And it's going to settle your configurations.

Jake Bernstein: Yeah. And it sounds like an assault on Shadow IT.

Kip Boyle: Yeah, it kind of is. So, there are some political ramifications to application whitelisting, but I think the business case for it has never been stronger. And reach out to me if you want to know more about it, I'm happy to talk with you. Let me give you one more Essential Eight practice, which is two-factor authentication. And I think most people know that this means your user ID, your password, and then something else, a fingerprint, a face scan, or a code that you then supply-

Jake Bernstein: Preferably not via SMS.

Kip Boyle: Yeah. You don't want a text message based code if you can help it. I mean, it's better than not having two-factor authentication, but text-based codes can absolutely be manipulated in a way that other forms of second factor authentication can't.

Jake Bernstein: Yeah. Okay. So, what about an attack that uses a so-called zero-day exploit.

Kip Boyle: Ah, okay. So, there's the zinger. Okay, thanks. A zero-day exploit. So, for anybody who doesn't know, a zero-day exploit means a flaw in a system that is not publicly known, but that an attacker does know about. And we know that governments are stockpiling zero-day exploits. We know this because one of them was actually stolen from the NSA and it's called eternal blue. And it was the basis upon which the NotPetya and the WannaCry cyber attacks were based on causing billions of dollars of damage in 2017. So, zero-day exploits are an issue. What I love about the Essential Eight though, is even if somebody comes at you with a zero-day exploit, the Essential Eight is going to severely strict the ability of the attackers to take control of your systems through things like two-factor authentication for your administrative accounts. So, even if I steal your user ID, and your password for an admin account, unless I can get my hands on that second factor of authentication, my zero-day exploit is not the magic pixie dust that it would otherwise be if I wasn't using two-factor authentication.

Jake Bernstein: Yeah, exactly. All right. So, let's bring this back to the original question here, which is, how are these attackers manipulating the MSP trusted relationship with our listeners, and what should our audience do to protect themselves?

Kip Boyle: Right. So, here's the whole point. And we had to go through that dialogue in order to make it clear to everybody kind of what we're talking about, and how we have gotten into a situation where we're using MSPs, and all of a sudden, we've maybe woken up to the fact that we are vulnerable. Well, for first of all, as we mentioned to using MSP, or an MSSP, or a CSP Cloud services provider, you have to establish a trusted relationship. And that trust comes with great benefits, but it also comes with this potential downside. So, you're very much at the mercy of the MSP that you choose to do business with. And most people are using more than one MSP, more than one Cloud Service Provider, more than one managed security solution provider. So, you got to look at the whole picture there. And I think the answer comes down to a combination of technical, and non-technical approaches. So, having you as co-host is fantastic because contracts actually make a difference here. And so, please, tell us what do you want to do with your contracts, Jake?

Jake Bernstein: So, I think with... There's a lot of things that need to be done here to protect yourself. One of them is to really think about what kind of access you have to give an MSP in order for them to do their job. Obviously, some of these, they have the keys to the kingdom, or they actually are the castle, but that's not always the case. So, step one is practice good base level security concepts like lease privilege, and separation of duties. If you don't need to have every MSP have full admin access to your system, don't do that. And then in terms of your contracts, you'll want to tailor your kind of level of contract to the level of access and risk that the MSP poses. In my practice, I generally recommend a three tiered approach with the highest tier being an MSP that does have the keys to your kingdom.

Kip Boyle: Yep.

Jake Bernstein: Then the lowest year being one where... They almost don't have access to data. And what this does is, one, it can really speed up your contracting process, but two, it also ensures that you are thinking critically about what your MSPs are doing, and what the contracts require them. And then-

Kip Boyle: You said something, hold on, you said something really interesting. You said it can speed up the contracting process. And I think most people are thinking that this would actually slow it down. So how does it speed it up?

Jake Bernstein: Well, so, if you have, let's just say, if you try to have a one size fits all contract, then every single MSP that you use is going to go through something that's either too stringent, or not stringent enough. And so, if you've got three tiers, then you can... Not only does it force you to make the decision early on about what contract to start with, it also... The less complicated contracts simply get reviewed faster.

Kip Boyle: I see. Okay. So, there's less redlining?

Jake Bernstein: Yes.

Kip Boyle: Yep. Okay. So, not as many iteration.

Jake Bernstein: Yep.

Kip Boyle: That makes sense. Okay. Thanks for clarifying.

Jake Bernstein: Yeah, no problem. And then, I think the rest of it is your typical third party risk management type things that you have to do. One of the things I think is really important when you're choosing an MSP from the legal side is, just because you have a contract, and that contract has in indemnification language, you got to keep in mind the end game here, which is always ask, is there enough insurance or enough money to actually make you whole, if the MSP gets breached, or is the vulnerability?

Kip Boyle: Yep.

Jake Bernstein: And this is a major, major consideration that I think a lot of people kind of forget to wonder about.

Kip Boyle: Mm-hmm (affirmative).

Jake Bernstein: And you know what? If you're contracting with Google, AWS, or Microsoft, it doesn't matter. You don't necessarily have to think about it, those are some of the biggest companies in the world. On the other hand, there are a lot of MSPs, and SSPs that are way smaller.

Kip Boyle: Mm-hmm (affirmative).

Jake Bernstein: They may not have resources. So, if you're signing up with someone, and relying upon your contractual obligations for protection, you need to make sure that your partner there has the ability to actually pay up if they need to.

Kip Boyle: Right. And the price points for a lot of these MSPs and SaaS providers, you're paying a very small amount of money. And so, if some really huge catastrophe happened, first of all, contractually, if you're going with one of the big three, they're going to give you a service credit, which is not even going to come close to covering the cost of the fumble of the ball, right? I mean, it could cost you the game, and you're going to get a service credit of 10% of your monthly fee, which isn't that much.

Jake Bernstein: Well, and just to be clear, when you're contracting with one of these, a big player, you're probably having to use their contracts, not-

Kip Boyle: Right.

Jake Bernstein: And-

Kip Boyle: Well, they're going to have a term of service, right? It's going to be a shrink wrap, or like a click.

Jake Bernstein: A lot of the times it will be. Now, it really does depend on the situation, and what you're doing, but a lot of these big providers have, I like to call them shared responsibility model of liability. And this is a common issue with Cloud service in particular.

Kip Boyle: Yeah.

Jake Bernstein: Make sure you understand where their responsibility ends, and yours begins.

Kip Boyle: Yeah. Now, this also reminds me that I should mention again, because you probably heard this on a previous episode, but I have a new course out in the LinkedIn Learning Library about outsourcing security. And in there, I talk very specifically about contracts, third party cyber risk management, things that you need to do if you're going to outsource security. So, if you want to really explore or this in depth, that's one place that you can do it.

Jake Bernstein: Yep, exactly. And then of course, also, as we already said, you should adopt the Essential Eight yourself. Obviously, the MSP should do it, but also, the customer should do it as well.

Kip Boyle: Absolutely. If you use an MSP, and you are using the Essential Eight, even if the MSP gets compromised, and malicious code gets sent down the wire to you, your use of the Essential Eight is going to provide a very strong protection against that. So, you should absolutely be considering Essential Eight. And by the way, the NotPetya cyber attack from 2017 is a wonderful one to study on this, because that's exactly how NotPetya got kicked off, was a Ukrainian software as a service provider was compromised. And in the course of receiving a routine software update that was pushed out over a trusted channel between the software publisher and the people who were using the software, is how that all got started. So, we've absolutely seen this in a big way in the real world. Well, Jake, any other... Any final comments before we wrap?

Jake Bernstein: No, I think that covers it.

Kip Boyle: Excellent. Today, we talked about how cyber attackers are exploiting the trusted relationship between IT service providers and their customers. And we talk talked about what you should do about it. We'll see you next time.

Jake Bernstein: See you next time.

Speaker 1: Thanks for joining us today on The Cyber Risk Management Podcast. Remember that cyber risk management is a team sport, so include your senior decision makers, legal department, HR, and IT for full effectiveness. So, if you want to manage cyber as the dynamic business risk it has become, we can help. Find out more by visiting us at cyberriskopportunitiesd.com, and focallaw.com. Thanks for tuning in. See you next time.