Today I want to share with you some common mistakes companies make when measuring the effectiveness of their security Awareness programs, including relying on a single piece of information, easy-to-detect phishing assessments, and creating one-size-fits-all tests. Let me help you build a quality program that actually works.
Watch the full episode below or listen on Apple/Spotify Podcasts. (links to Apple and Spotify on the Gone Phishing page!)
Episode 7: Listen on Apple
Full Episode 007 Transcript:
00:00:00:12 - 00:00:34:21
Connor Swalm
Welcome to Gone phishing, a show diving into the cybersecurity threats that surround our highly connected lives. Every human is different. Every person has unique vulnerabilities that expose them to potentially successful social engineering. On this show, we'll discuss human vulnerability and how it relates to unique individuals. I'm Connor Swalm, CEO of Phin Security, and welcome to Gone Phishing. Everyone in is Connor CEO at Phin and thank you again for joining me on Gone phishing.
00:00:35:02 - 00:00:56:00
Connor Swalm
So we've dived into a couple of topics that are near and dear to my heart. And today we're going to jump into measuring the success of your security program correctly. So a couple of things that I'd like to state. I've watched hundreds and hundreds of security awareness programs launched both an individual companies and in the partners that I personally serve.
00:00:56:04 - 00:01:14:06
Connor Swalm
I've seen some people do it incredibly well. I've seen some teams do it amazing, and I've seen some people do it. Not so well. So I figured I'd distill for y'all a little bit of what are the commonalities I see between people measuring and running programs effectively? And then what are some of the common ways that people don't?
00:01:14:14 - 00:01:33:00
Connor Swalm
Now, I know that we've talked in one of the previous episodes about some of the pitfalls. So actually what I'm going to go into today is a topic that I addressed in a blog actually. It's called Measuring Phishing Incorrectly. It's on our website. And there are four specific areas that I really believe that some people do things incorrectly.
00:01:33:00 - 00:01:55:16
Connor Swalm
So I wanted to address those today. So the first common way that I see where programs are measured incorrectly is a very common thing that we're all somewhat guilty of in various areas of our life. It is taking one piece of information and applying that to the entire outcome. So simply put, one piece of information is not enough.
00:01:55:16 - 00:02:26:00
Connor Swalm
In the same way, you wouldn't measure your health by just your blood sugar or your heart rate or your blood pressure. Now, all of that is very important and could help paint a whole story about the actual overall nature of your health. But just any one piece of information isn't going to tell you everything. So in the same way, that kind of what I see is companies will take the overall phishing percentage and say, you know, it's not just companies, it's cyber insurance carriers, cyber insurance brokers.
00:02:26:00 - 00:02:49:18
Connor Swalm
Everyone wants to see a, you know, the coveted. And if you're not watching me, I'm quoting here the coveted 0% phish rate. And if you get anything above that or anything, you know, north of, let's say, 7%, then your program is not effective. And that's just simply not accurate. There are many ways to measure the effectiveness of your program, and really what you need is a whole perspective.
00:02:49:19 - 00:03:09:06
Connor Swalm
Not only is phishing performance or the ability for a human to recognize social engineering going to be a piece of it. But other things such as the general sentiment of the program, the engagement, how many times did the managers have to follow up with a user to tell them, Hey, you're not doing something and we need you to go do that or you're not completing your training.
00:03:09:10 - 00:03:35:03
Connor Swalm
So there's a lot of different things that need to be taken into account when measuring the effectiveness. So don't take one piece of information and apply that to the entire outcome. It really requires a couple of different pieces of information for you to judge the correct effectiveness of your program. The second is the second thing that I see people do or some programs run incorrectly is phishing assessments that are easy to detect.
00:03:35:03 - 00:04:02:16
Connor Swalm
Now, before you go off on me, whoever's listening to this and say, well, phishing assessments aren't really valid or don't really have any benefit, I will pose this. The goal of phishing assessments is kind of like a scrimmage for a game for a baseball game and scrimmaging. Absolutely. Has time and a place for teaching people to be effective or teaching people how to play the game, so to speak.
00:04:02:22 - 00:04:26:00
Connor Swalm
So I've always viewed social engineering assessments or phishing assessments because that's the most common form of social engineering as an opportunity for a person to demonstrate how they'll perform in real life. Now, people's vulnerability changes. The things they're vulnerable to will be different. The people they work with are going to change always and consistently. You know, change is the only consistency here.
00:04:26:00 - 00:04:50:14
Connor Swalm
And so what I've put here is that if you're phishing assessments are incredibly easy to detect in this specific way, that the way the phish is delivered, the way the social engineering is conducted, means the employee doesn't need to recognize that something is actually a social engineering attempt. They just recognize the domain that they've seen in the past that emails have been sent from or the poorly crafted wording or whatever it is.
00:04:50:18 - 00:05:10:10
Connor Swalm
There are ways in which you can send the actual assessments that tell the employee it's an assessment without needing them to think through it. And that's what we're really trying to get to, is that an employee is able to recognize and understand on their own what phishing is going to look like for them specifically and what they might, may or may not be vulnerable to.
00:05:10:10 - 00:05:28:19
Connor Swalm
So don't make phishing assessments easy to detect just by the nature of the way they're sent. Make them actually a test. You know, you need your test to actually test people. So if you deliver it in such a way that it doesn't test anyone, then you're not getting any new information. You're actually generating information that's not usable in some way, shape or form.
00:05:29:06 - 00:05:52:02
Connor Swalm
And the second one, not the second. The third thing that I see is one size fits all phishing tests. And this is a concept that keeps coming up in our conversation about human vulnerability management is that every employee is unique. One employee is going to be vulnerable to an Amazon gift card scam, whereas another employee is going to be vulnerable to business.
00:05:52:02 - 00:06:13:10
Connor Swalm
Email Compromise. Actually, we just went through this as a company. We had some new employees that just got a phishing attempt, realistic. Somebody texted them and said, Hey, this is Connor. So they were impersonating me and they were saying, I need you to join this meeting for one of them. And the other was, I need you to go buy us prepaid gift cards for a big deal that we're trying to close.
00:06:13:10 - 00:06:34:00
Connor Swalm
And fortunately for us, both of those employees immediately contacted me via other means and I'm like, Hey, is this actually you? And do we actually need to do anything together? And I said, Absolutely not. And now we warn every new employee about the impending phishing attacks. But what was really interesting is that they were different phishing assessments or not assessed actual phishing in this case.
00:06:34:08 - 00:06:57:07
Connor Swalm
And if a different phish had been delivered to the different employees, maybe we would be talking about a different scenario that it occurred. So what I always try to do and when when I'm helping other companies launch actual awareness training programs, of which phishing is a piece of it, I'm trying to get to the bottom of what are the people in their company actually vulnerable to?
00:06:57:12 - 00:07:29:06
Connor Swalm
And you could do that by setting, you know, overall topics for the specific vertical vertical that clients in. So, you know, maybe sending Amazon scams to a banking client is not going to be effective, but sending emails from zits and if you're in banking, you know what's access might might be more way more effective and also more representative of what they're likely to come across in real life while they're doing their job, which is the goal to prepare them for reality, you need to make sure that every employee is getting the experience that is most beneficial to them.
00:07:29:13 - 00:07:49:17
Connor Swalm
That is incredibly hard to do. So I'm not saying I'm not saying that you have to do that in order to have any amount of effectiveness. I'm just saying that's what we should be aiming for and that's what we should be approaching. And the fourth and the fourth way that I see programs running correctly, measured in correctly, is something near and dear to my heart.
00:07:50:08 - 00:08:14:03
Connor Swalm
It's math. And so what I mean by this is, let's say you run security awareness training program and you have an overall phish rate of 4% every single month. Well, first off, you know, see, point number one, don't use one piece of information to judge the effectiveness of your entire program. But let's say you are doing that in this case, so let's ignore that for a bit.
00:08:14:03 - 00:09:02:03
Connor Swalm
If you had a 4% phishery in each of those three months, but they were 4% completely unique people, groups of people in your company, what you really have is a 12% efficient rate for that quarter. So now if you're measuring per year, that number could be huge. One thing that is really important for us to understand, not only as the awareness training experts here, but also as the company who's going to be conducting these exercises is what percentage of your client base is vulnerable to one, to phishing, and then to who are the people that are just repeatedly demonstrating that they need additional support, that they are continuously falling for scams that should be recognized
00:09:02:03 - 00:09:25:22
Connor Swalm
or that are commonplace, that they are they're clearly missing some kind of education or some kind of warning system in their in their brain, in the patterns, in the way they behave that would help them recognize what's going on. So I don't have an incredible amount to say today about how you can interpret the results of the programs correctly.
00:09:25:22 - 00:09:54:11
Connor Swalm
But what I would say is, let's go through all of these points and let's just take the opposite and we'll be in a really good spot. And this is kind of what I do with a lot of the partners that we work with on the companies we work with, is let's identify multiple pieces of information that are important for us to track, one of which might be phishing percentage, one that I absolutely believe and recommend that every company began tracking is the rate at which your employees actually engage with the training content on a consistent basis.
00:09:54:18 - 00:10:14:05
Connor Swalm
So if you're delivering training consistently, how many times, how many people does your manager or the director report for? Those individuals have to get involved and say, you need to go complete your training. The lower that is, the higher rate that people engage the way more supportive people will be of your program, of your security team in general.
00:10:14:18 - 00:10:38:19
Connor Swalm
The second is this phishing rate go up when you start to actually assess people at a unique level. And when you don't make it easy to detect one very simple way, one thing you can take home and do today is do not deliver the phishing assessment for, quote unquote, the month or the quarter at the same time of day to every single employee at the same phish.
00:10:39:02 - 00:11:00:22
Connor Swalm
That is number one. You know, people should be able to do what I call prairie dogging, where they just stick their head up and say, hey, is this a test for the quarter? And then immediately get rid of it? They need to be able to recognize social engineering. And that kind of solve point three as well is don't make it one size fits all, craft it, you know, even if it's random randomization is way better than giving it to everybody initially.
00:11:01:14 - 00:11:29:01
Connor Swalm
So the fourth poorly interpreted results really get to the bottom of who is generating the data that you're looking at. So you need to see who is getting pinched, who is clicking on the things and who is actually completing their training. And you need to look at that from an individual perspective so that, you know, you've identified the group of people that are actually vulnerable and more importantly, that need additional support.
00:11:29:10 - 00:11:47:21
Connor Swalm
That's the most important piece. But also you're able to rule out the people that fell for a single phishing assessment and now they're good. Now that training might be working, or now that they're more visual, maybe they're more vigilant, which happens a lot when people start getting phish. So you just need to know who needs more support and who is doing their job great.
00:11:47:21 - 00:12:04:09
Connor Swalm
Or who you know, gets this on the first try so that you are fine. So with that, those are all of the again, I would highly recommend go read this blog Measuring phishing and frankly you can find it on our website. I highly recommend you take a look at that and reach out to us if you have any questions.
00:12:04:22 - 00:12:22:16
Connor Swalm
That is what I've been gathering over the course of these last several years in helping run and launch and manage many, many programs. So these are the most common ones that I see and really just take the opposite of these four incorrect measure ways that people measure. And you will get to a really high quality security where this program.
00:12:23:01 - 00:12:45:12
Connor Swalm
So next time and often what we're going to talk about is how to apply human vulnerability to your security program. So that level up, so to speak, of a security awareness training is what I call human vulnerability management. So how can you actually kind of craft some of that into your security awareness training program? Or how can you create a real human vulnerability management program?
00:12:45:18 - 00:13:04:19
Connor Swalm
That's what we're going to get into next time. So I'd love to see you there. I'm Connor, CEO Phin and we'll talk soon. Thanks so much for tuning in to go on phishing. If you want to find out more about high quality security awareness training campaigns, how to launch them in ways that actually engage employees to change their habits.
00:13:05:02 - 00:13:19:22
Connor Swalm
Then check us out. Phin security at phinsec dot IO that's an assessment that IO or click all of the wonderful things in our show notes. Thanks for visiting with me today and we'll see you next time.
