I’m joined again by Wes Spencer from Fifthwall who I invited back to share with us a real-life cyber security breach story that cost the company hundreds of thousands of dollars plus some public embarrassment. So grab your marshmallows and join us around the campfire and let's learn from someone else's mistake.
Watch the full episode below or listen on Apple/Spotify Podcasts. (links to Apple and Spotify on the Gone Phishing page!)
Episode 9: Listen on Apple
00:00:00:12 - 00:00:23:21
Connor Swalm
Welcome to Gone phishing, a show diving into the cybersecurity threats that surround our highly connected lives. Every human is different. Every person has unique vulnerabilities that expose them to potentially successful social engineering. On this show, we'll discuss human vulnerability and how it relates to unique individuals. I'm Connor Swan, CEO of Phin Security, and welcome to Gone phishing.
00:00:28:12 - 00:00:51:05
Connor Swalm
Hey, everyone and welcome back to Gone phishing. I'm Connor and today I am joined once again by the awesome the great Wes. Spencer Wes was on the show just a few episodes back and I wanted to have him back again. To talk more specifically about a recent cyber breach. He told me while we were getting ready for this that he had an awesome story.
00:00:51:05 - 00:00:58:14
Connor Swalm
So I'm actually really excited and if there's anything, we can all learn from it. So, Wes, without further ado, welcome back. How are you.
00:00:58:14 - 00:01:06:10
Wes Spencer
Today? Thanks for having me, Connor. I'm doing great. Just great. Phenomenally great. Better than the folks in the story I'm going to share. How about that?
00:01:07:22 - 00:01:11:18
Connor Swalm
I feel like if you said good, you'd still be doing better than the folks in the story. You're going to be.
00:01:11:20 - 00:01:13:15
Wes Spencer
So I would think so.
00:01:14:01 - 00:01:21:02
Connor Swalm
But that's awesome. I'll let you do it then. So you said you have an interesting story for us. I'll let you tell it.
00:01:21:07 - 00:01:24:21
Wes Spencer
Well, did you say something about marshmallows before? Do I have to pull the marshmallows around?
00:01:25:03 - 00:01:36:16
Connor Swalm
Yeah. So I was just going to get marshmallows out and roast them over the fire while you tell this story. Because I feel like it's going to be either a spooky ghost story or it's going to be wonderful and relaxing. Like I'm camping in the summertime.
00:01:36:18 - 00:01:51:06
Wes Spencer
Yeah. Don't you love those? Like, yeah, these dark, scary campfire stories, by the way, breach stories are like that. Like, no one ever wants to hear what I do in security, like, around the family table, like, oh, boring, whatever nerd stuff. Until they have a breach story, they're like, wait, why tell me more that really happen? Are you kidding?
00:01:51:14 - 00:02:12:02
Wes Spencer
So I'll tell you one. This is a fairly recent one. So MSP that I know I'm going to no names will be shared. They they were in, you know the good midsize MSP nothing special about them and they were courting a very large client who had a local branch inside their main home. There, their main like town they work out of.
00:02:12:07 - 00:02:27:18
Wes Spencer
But the client was quite large, like really, really big. They just happened to have like a, we'll just call it a factory that was there. And so they've been having some trouble at their local i.t and we're looking to make some changes. And so they talk to the MSP and they're like, Oh, we'd love to help you. We can do everything you want.
00:02:27:18 - 00:02:46:04
Wes Spencer
We're great at this. And so what was interesting was this this client was by far the largest client that they had ever taken on by an order of magnitude bigger. Right. And so the MSP was like super pumped about this because from their point of view, they're like, whoa, look at the new revenue we're going to get out of this.
00:02:46:04 - 00:03:01:17
Wes Spencer
And we called them this crazy number and they were like, Yeah, no problem, let's make this happen. This sounds great. I guess from them it was cheap. And so they went in. They actually hired a dedicated account manager for this role. They hired two or three tax to be able to support it. And they were like, I mean, totally.
00:03:01:17 - 00:03:16:21
Wes Spencer
They bit off more than they can chew 100%. But they, you know, what could possibly go wrong? That kind of money coming in will go fix any problem or throw money at the problem. And what could we possibly have? Right. So everything's going fine for a few weeks. But the onboarding took a really long time for the MSP.
00:03:16:21 - 00:03:39:20
Wes Spencer
Like they just the worse is slow to get them going because they're working with the local branch and the headquarters. It's further away is just it's a lot more work to kind of get everything filtered through. And so over time they slowly started to get things kind of put up in place. They did get an ETR rolled out and the ETR was there and it was it was basically alerting and things like that.
00:03:40:03 - 00:03:46:21
Wes Spencer
Well, what happened was late one night and I kid you not, it was during a very big sports ball event is all I'll say.
00:03:47:05 - 00:03:50:04
Connor Swalm
But it's interesting how it all happens at times like that, right?
00:03:50:04 - 00:03:51:22
Wes Spencer
Yeah, it always is. These times, yeah.
00:03:51:22 - 00:03:53:03
Connor Swalm
The infamous three day weekend.
00:03:53:08 - 00:04:11:21
Wes Spencer
That's right. Yep. If you know what we're talking about, the July 4th a couple of years ago. So this is a big sports ball week and it happened to be a Sunday. So you can probably guess what the event was. And the account manager for the client got a call during the middle of a sports ball game and she's talking to him like, Yeah, you know, nothing's online right now and I think we're okay.
00:04:11:23 - 00:04:26:10
Wes Spencer
It might just be some kind of like bigger like outage with our headquarters, but we've got some of the folks in the Middle East that are coming online for our our factories. Just wanted to see if you guys are aware of anything like. Well, I don't know if you click, click, click, click, click. Yeah, no, everything's good on our end.
00:04:26:18 - 00:04:41:21
Wes Spencer
But if you can't reach anything, that's strange. They go and they try to go into the room and remember the tool they use to connect and do things. So they connect. And sure enough, everything's offline over there. They can't see anything either. And so now they can't connect to those machines. They're not available online. It's you know, they were trying to, like, flail around, like, is this an outage?
00:04:41:21 - 00:04:58:01
Wes Spencer
Like an Internet outage? That's what they thought it was at first. And so they try not to panic client because they're still onboarding them, still getting everything going. And they but the account managers smart enough to say, I'm going to call our president and just let him know what's going on. And so he was already like pretty inebriated in the middle of the game.
00:04:58:01 - 00:05:14:02
Wes Spencer
It's super loud. And he's like, Yeah, I just try to take care of this. Whatever you need, call me back. You need it, but, you know, whatever. And so, you know, she he hung up with her. And so they're still flailing around. She gets a couple of like the local I.T. guys that are on call. They're not like security people trying to figure out what's going on.
00:05:14:05 - 00:05:37:06
Wes Spencer
Those jokers reset the army. I'm trying to figure out what was going on. And that was a big mistake because then they fully lost their connection and they lost a ton of data. If you don't know when you restart a machine, you lose a lot of the forensics data that was there. So sure enough, finally, some of the Middle Eastern folks start coming online and they start calling in because all of the computers are not accessible.
00:05:37:06 - 00:05:56:13
Wes Spencer
And there's now this text file, it's on every single desktop and they open it up and it's a ransomware note and it says, We've encrypted all your files, all your machines. So now it goes red hot. Right now they are very angry because they're losing millions and millions of dollars that they're about to. And you're talking about a company that's got offices all over the world.
00:05:56:13 - 00:06:11:12
Wes Spencer
And so they are now like, you will fix this problem. What's going on? How could you let this happen? Blah, blah, blah. So the MSP is flailing. They have no idea what's going on. So she calls the CEO up, the CEO of the MSP and says, Hey, this is bigger. It wasn't an outage. It looks like it's ransomware.
00:06:11:12 - 00:06:27:23
Wes Spencer
He's like, What are you guys doing fixing it? And so they're panic. They finally bring their security person from the MSP onto the call, and he's normally like, this really good guy, totally understands cool under firewall. He panics and cracks under the pressure because he got brought into it. It's late at night and he was probably had a few drinks himself.
00:06:28:05 - 00:06:46:09
Wes Spencer
And so long story short, this thing went really, really south and they couldn't figure out how to fix that. They couldn't figure out how to bring anything back up online. They were getting screamed at from the from the client. Because of all this money they're losing, they're totally out of no ability to know what happened. So what what ended up happening was they fight.
00:06:46:09 - 00:07:08:15
Wes Spencer
They didn't have cyber insurance. So they finally had to just pull in with a ton of money at the cost of the MSP and the client digital forensics firm who came in, started assessing the situation, discovered it was actual ransomware. Everything had been encrypted. And the problem was and this is where the darkness gets the whole thing is they were still in the middle of onboarding konnor so they didn't have backups made none at all.
00:07:08:18 - 00:07:29:05
Wes Spencer
There was nothing and it was because it was a handover. And then come to find out this is where it also gets nasty is there? They were only one part of the solution provider for all of the i.t. Presence this big company had. They thought they were doing everything well. Come to find out a lot of the other branches had their own i.t.
00:07:29:12 - 00:07:52:05
Wes Spencer
And so they walked into the middle of a situation. They never bothered to ask the questions to the client of like what is your entire corporate i.t look like? Who manages what, what's our responsibility? They just ramp right into the situation. They're they're emcees in the in the aftermath of the breach were really bad they didn't dictate when they became a client versus when the onboarding would finish.
00:07:52:10 - 00:08:10:17
Wes Spencer
So they ate a ton of cost on their own liability. To get everything up and running, the client had to pay the ransom demand, but the MSP lost tons and tons and tons of time and resources. Money on the full imaging restoration. Reinstalling everything ended up being a colossus. Of course, they fired the MSP at the end of it.
00:08:10:17 - 00:08:30:13
Wes Spencer
Right? So not only did they get the public embarrassment, not only did they get a ton of revenue they thought were going to make, their's goes downhill fast. They burned thousands of people hours into the response and recovery of all of this, that all of this could have been avoided had they had much clean and clear like onboarding processes and really steering them through a methodical process to build this out.
00:08:30:13 - 00:08:53:05
Wes Spencer
They didn't do any of that. And so just in their hunger and rush for money, they completely set their company back at least a year in terms of revenue. So an unbelievable story. And, you know, you just watch this firsthand and you're just like, I can't believe this is happening to you guys. Like what is going on? And by the way, I'm sure all this started with a human right, because I'm sure it came to see from the forensics.
00:08:53:06 - 00:09:10:06
Wes Spencer
I'm sure it came from a phishing email that they got directly in. And the other thing, too, that stuck out to me and I'll just finish on this, is the fact that the account management people led this whole thing from the very beginning. Like where were the security people with the second they said, Hey, this things are offline.
00:09:10:06 - 00:09:23:18
Wes Spencer
I'm not saying it's a breach, but can someone take a look at this or an incident? No, it was the account management people that were just more focused on saving the account than it was doing the right thing to limit the blast radius before things got really, really bad.
00:09:24:20 - 00:09:52:02
Connor Swalm
Yeah, it it definitely is a concept you and I talk about a bunch, but you first told me about it a while ago, which is having a breach occur. It's no longer an if, it's a win conversation and it feels like if that would have been the pervasive mindset from everybody MSP side, client side, that there would have been a lot of things done differently.
00:09:52:02 - 00:09:57:22
Connor Swalm
What are some of the things you think that we could learn from that incident? I could I could think of a couple, but.
00:09:58:00 - 00:10:17:23
Wes Spencer
Well, I'll piggyback on what you just said. MSP or. Well, waking up to the fact that you have to operate like this is going to happen to you at some point, which means have you had a good attorney review your MSAs, your master service agreements? Have you actually worked through and built a good incident response program? Have you communicated that with your client?
00:10:18:00 - 00:10:35:13
Wes Spencer
And are they under the same level of understanding as you are with incident response and knowing what you do, what they do do? Does the client have cyber insurance? Do you have what's called tech, you know, insurance for yourself? Are you educating your client all the way across this journey? Because we just don't do that enough and we think we can do it.
00:10:35:13 - 00:10:42:22
Wes Spencer
When something happens, it hits the fan. And that's when things get way, way, way, way worse because we're not operating on the same wavelength at all.
00:10:43:11 - 00:10:57:18
Connor Swalm
Yeah, it's I educate a lot of MSPs today is the questions you don't ask are more damaging than answers to the questions you do? Yes. Is that one of the stories your kids actually like to hear that are table or was that did then not get past the bar?
00:10:57:18 - 00:11:09:07
Wes Spencer
For that I have to spice it up even a lot more than that. Just to get them to be even remotely interested. I have to give like the hacker a name and you know, they've got to be, you know, some kind of fight and have to go like fly and duke it out with them, things like that, you know.
00:11:09:07 - 00:11:11:13
Wes Spencer
So they do spice it up a little bit for the kids, but hey.
00:11:12:07 - 00:11:18:07
Connor Swalm
Us, any last minute pieces of advice, anything you'd like to say to the folks before we have off here?
00:11:18:10 - 00:11:33:04
Wes Spencer
Just stories are good because they really help you understand what could happen and clients should know. It's good to have an augmentation of like a bunch of stories stored up in your brain that you can always tell clients when it's time to talk because everyone loves a good breach story. So yeah.
00:11:33:21 - 00:11:54:15
Connor Swalm
Yeah that there were binds of when I would what's happening in cyber city what I saw happening in cybersecurity is we're talking to people about a concept cybersecurity in ways that they don't understand about things they don't feel like they should care about. And I felt the exact same way when I would talk about the math I enjoyed doing in college.
00:11:54:15 - 00:12:18:17
Connor Swalm
So I can see that I can definitely see us benefiting from talking similar languages. So and thanks for that story. By the way, you didn't quite have marshmallows over an open fire here, but it definitely felt felt like I was listening to a scary story. So thanks so much for joining me. If people wanted to reach out to you, whether that's to hear more wonderful stories for the vets just to connect and learn a little more, where would you like them to do that?
00:12:18:22 - 00:12:34:02
Wes Spencer
Yeah, a couple of places. If you want to meet, you need help with cyber insurance. Reach out to us fifth wall solutions dot com easy jump right in will help you with your MSSP and user whatever you may be will help you there. And if you want to follow me LinkedIn just Wes Spencer you'll all turn right up and same with YouTube.
00:12:34:02 - 00:12:37:22
Wes Spencer
Just Google me there or whatever you search for there. I'll come right up as well.
00:12:38:21 - 00:12:49:05
Connor Swalm
Wes Spencer, the self-proclaimed dictator of the channel, will be sure to put that in the show notes, and we'll put all of Weson's information in the show notes for all of you listening. But thank you so much for joining me today.
00:12:49:12 - 00:12:52:08
Wes Spencer
Thanks, donors. A pleasure.
00:12:53:11 - 00:13:15:02
Connor Swalm
Thanks so much for tuning in to Gone phishing. If you want to find out more about high quality security awareness training campaigns, how to launch them in ways that actually engage employees to change their habits. Then check us out. Find security at phinsec.io
00:13:15:12 - 00:13:19:15
Connor Swalm
Thanks for visiting with me today and we'll see you next time.
