Are phishing simulations actually making employees more vulnerable to attacks? A brand-new study from UC San Diego suggests just that, challenging conventional wisdom about cybersecurity training. In this post, we'll examine the study's findings, explore its limitations, and discuss how organizations can rethink their approach to phishing simulations so that they lead to their desired security benefits.

This isn’t the first time the effectiveness of phishing training has been questioned. In my last post, I referenced critics like Matt Linton from Google, who, in his article "On Fire Drills and Phishing Tests," argued there’s no evidence these programs reduce phishing success rates, advocating for 'fire drills' instead. Similarly, security expert Bruce Schneier has expressed concerns, suggesting such training can sometimes backfire.

But armed with this fresh data, it’s time to look beyond the headlines and focus on the potential for improvement. What happens when employees face these simulations, and more importantly, what can we do to make them work better? Let’s dive into the possibilities.

The Study

Researchers from UC San Diego, the University of Chicago, and UC San Diego Health conducted the study "Understanding the Efficacy of Phishing Training in Practice." Over eight months, they scrutinized the outcomes of ten simulated phishing campaigns involving more than 19,500 employees at a large healthcare organization. This extensive research offers valuable insights into the real-world effectiveness of common phishing training methods and underscores the need for a more comprehensive approach to cybersecurity training.

Here’s what the study uncovered:

1. No Significant Impact of Annual Training
   Employees who recently completed annual cybersecurity awareness training were just as likely to fail phishing simulations as those who hadn't, indicating no significant correlation between training recency and effectiveness. This raises questions about the long-term efficacy of these once-a-year programs.

2. Minimal Benefit from Embedded Phishing Training
   Embedded training provides instant feedback during phishing simulations, controversially interrupting their workday. While it slightly reduced failure rates, the difference was marginal—just a 1.7% improvement compared to the control group. This finding suggests that even targeted, real-time interventions may not be moving the needle as much as hoped.

3. Low Engagement with Training Material
   Engagement with training materials was alarmingly low. Over half of the employees spent less than 10 seconds interacting with the follow-up training after failing a simulation. Worse, fewer than 24% of participants completed the training.

4. Potential Negative Effects of Repetitive Static Training
   Surprisingly, the study found that employees who completed more training sessions were more likely to fail subsequent phishing simulations in some cases. Specifically, for each additional static training session an employee completed, there was an 18.5% increased likelihood of failing future phishing attempts. This counterintuitive result suggests that repetitive, static training could lead to disengagement or overconfidence, making employees less vigilant over time. However, interactive training did not have the same adverse effects. More on that below.

Not So Fast: Major Gaps in the Study

The UC San Diego study paints a sobering picture of phishing training, but before we jump to conclusions, let’s look closely at its blind spots. Like any research, this study's scope and methods shape its findings—and there’s more to the story.

1. Click Rates Alone Don’t Tell the Whole Story

The study focuses heavily on click rates as a measure of success or failure. But phishing isn't just about clicks. What happens after someone clicks a link matters even more:

• Did they enter their credentials?

• Did they report the phishing attempt?

• Did they recognize something was off and stop midway?

These behaviors are just as critical but aren’t reflected in the study’s results. Click rates alone can’t capture the complexity of human behavior in phishing scenarios. As an industry, we must move away from clickthrough rates and focus on analogs for a successful attack.

2. One Organization, One Industry

First, this study was conducted within a single healthcare organization, which is a huge limitation. Healthcare has its own unique challenges—fast-paced environments, overburdened staff, and heavy reliance on IT systems. But does this mean the results hold in industries like finance, tech, or education? Not necessarily.

Even within this one organization, there’s a little breakdown by job role. Are IT staff just as susceptible as clinical personnel? Would a systems engineer react like a nurse would while handling patient emergencies? These nuances are left unexplored, leaving us with more questions than answers.

3. Is the Training Itself the Problem?

Let’s talk about the elephant in the room: training design. The study mentions that most employees spent less than 10 seconds on the training materials, and only 24% completed them. That’s not great.

But here’s the thing—what does that tell us about the training itself? Were the materials engaging? Were they relevant to the employees' daily work? One-size-fits-all training tends to fall flat because it fails to connect with people on a personal level.

Imagine trying to teach someone to recognize phishing emails with a generic "Don't click links!" message while their actual job involves processing dozens of email requests daily. Effective training needs to account for role-specific behaviors and risks.

4. The Fatigue Factor

Interestingly, the study found that some employees became worse at avoiding phishing emails after completing multiple training sessions. That’s alarming, but it also raises a critical question: was this training repetitive and disengaging?

While the study found that employees who completed multiple static training sessions were more likely to fail subsequent simulations, it did not find evidence of training fatigue affecting engagement levels. The researchers noted that repeat training sessions did not show decreased completion rates, suggesting fatigue might not be the primary factor. However, the study doesn't fully explore other psychological factors, such as inherent disinterest or overconfidence, which could be vital to understanding why training might backfire.

5. Why Not Experiment with New Approaches?

The study sticks to traditional methods like annual training and embedded phishing tests, which are widely used but far from innovative. There’s a growing body of evidence that newer techniques—like gamification, microlearning, and immersive simulations—can drive higher engagement and better retention.

For example, imagine a phishing simulation that adapts to the user’s skill level, gradually increasing in difficulty as they improve. Or training that gamifies the process, rewarding employees for spotting phishing emails. These aren’t just theoretical ideas—they’re already being tested in forward-thinking organizations.

6. Beyond Metrics: The Human Element

Finally, there’s the cultural impact of training to consider. Even if click rates don’t drastically improve, training might encourage employees to report suspicious emails, think twice before sharing sensitive data, or feel more confident about cybersecurity.

The study doesn’t measure these intangible benefits that are critical for building a security-conscious culture. A training program that fails to lower click rates but succeeds in fostering vigilance might still be worth its weight in gold.

Identifying Vulnerable Employees: An Overlooked Benefit of Phishing Simulations

Phishing simulations aren’t just about reducing click rates; they’re also valuable for identifying employees most vulnerable to phishing attacks. This insight allows organizations to provide targeted training, implement additional safeguards, and allocate resources effectively. Vulnerable employees can receive tailored support, such as personalized coaching or more engaging training. At the same time, high-risk individuals can be safeguarded with stricter security measures.

Rethinking Phishing Training for Effective Security

The UC San Diego study challenges the effectiveness of traditional phishing training methods, revealing significant gaps that organizations must address. Reliance on click-through rates as the sole metric overlooks critical behaviors that occur after a click, such as entering credentials, reporting attempts, or recognizing and stopping suspicious activity. To truly enhance security, organizations must adopt more comprehensive metrics that capture the full spectrum of employee responses.

The study also highlights issues with training design and engagement. Low participation rates and the potential negative effects of repetitive, static training suggest that more than a one-size-fits-all approach is required. Effective training should be engaging, relevant, and tailored to employees' specific roles and risks. Exploring innovative methods like gamification, adaptive difficulty, and personalized content can improve engagement and retention.

Furthermore, phishing simulations offer the overlooked benefit of identifying vulnerable employees. By recognizing who is most at risk, organizations can provide targeted support, implement additional safeguards, and allocate resources more effectively.

In essence, protecting against phishing attacks requires a multi-layered defense strategy. This involves:

• Moving beyond click rates to assess training effectiveness.

• Standardizing and evolving phishing simulations to reflect real-world threats.

• Enhancing training methods to be more engaging and role-specific.

• Leveraging the full benefits of simulations to identify and support vulnerable staff.

Addressing these gaps can help organizations build a more resilient security culture that adapts to evolving threats. It's time to rethink our approach to phishing training—not abandon it, but make it smarter and more effective. Through comprehensive strategies and a commitment to continuous improvement, we can better protect our organizations and empower our employees to be the first line of defense against cyber threats.