The Value of a False Positive. Part Two: A New Definition

As a reminder, we better understand true positives in part one of our series. The true positive rate (TPR) formula is now the following:

TPR = (TP + TP:B + TP:AR) ÷ (TP + TP:AR + TP:B + FP)

With false positive (FP) being defined as:

FP = Total alerts – (TP + TP:AR + TP:B)

Paths of measurement and work

The new TPR and FP definitions above create the following paths of measurement and work:

• True Positive Rate (TPR): This is our reportable metric, broken down into other more granular insights such as:
  • Security Tool TPR: The performance of an individual security tool’s TPR (i.e., CrowdStrike TPR vs. Exabeam TPR),
  • SOC Analyst TPR: We can track analyst-reported true positives against our overall true positive trend to identify under and overperforming analysts.
  • Department TPR: TPR broken down by the organizational department to see if specific departments are “riskier than others.” Do they tend to generate more true positives than other departments?
  • Individual Rule TPR: The performance of a particular security rule. Does it generate a TPR that is less than the rest of our program? If yes, it’s a good candidate for tuning. This can also apply to the engineers who create the tools!
• False Positive (FP): This is our priority work. We aim to get this measurement as close to zero as possible. It is also super important to ensure we don’t have any true positives floating around in this number. Create a feedback workflow where senior analysts check false positives. 
• True Positive (TP): Our secondary priority is generating high-quality alerts that result in consistent true positives. I can’t stress enough that this is a measurement, not a metric. The goal here is to increase the overall COUNT of true positives. We can use the other measurements to improve our overall TPR.
• True Positive: Accepted Risk (TP:AR): There is not much to do here but put pressure on your exception program. Accepted risks can carry corporate and personal liability, so this should be well understood and have a chain of custody for known and accepted risks.
• True Positive: Benign (TP:B): This is our gray area that can be evaluated program-by-program. If you have the bandwidth to deal with a higher volume of alerts, having a higher TP:B will allow you better insight into your environment and fewer false negatives. Otherwise, this represents a tunable number. Determine which individual alerts contribute the most to this rate and tune from there. 

Ultimately, this isn’t meant to be an exhaustive guide to everything true positive vs. false positive. Still, it should be treated as a primer to help you consider how to use this reasonably powerful metric in your program.

An interesting poll

An industry colleague of mine, Josh Johnston, polled his LinkedIn connections not long ago to understand other people’s true positive rates in their security programs. He had 49 respondents across multiple disciplines, skill levels, and industries. Here is the poll as presented on LinkedIn:

 These are the results:

This (albeit limited) poll is starting to confirm one of my hypotheses about true positive rates. It’s normal for this rate to be incredibly low. My gut feeling is that most security programs have a positive rate of around 3% or less. If you start breaking down how many alerts you have any given day in your program, a 3% true positive rate would say that for every 97 false positive alerts, you get three that are actionable. Think about that: We have all had the tools that produce THOUSANDS of alerts a day… is a 3% true positive rate even close to realistic? I don’t think assuming some organizations have a less than <1% true positive rate would be too bold. A rate of less than 5% may even be the standard in the industry. 

Because Josh works for a vendor in the cybersecurity industry, he is likely to be connected with others who work for security vendors in his LinkedIn network. For fun, he broke the poll results down by vendor responses vs. non-vendor responses; the results made me chuckle a bit. 

Based on this poll, it wouldn’t be much of a stretch to infer that vendors tend to dramatically overestimate the number of true positives their tools produce. 

A call to action; recommended workstreams 

The following actions for security teams are recommended based on these revised definitions and observations.

• Add the new classifications to your case tools. 
• Calculate and report on the new actual percentage of the True Positives Rate and the more granular measurements for your environment.
• Use the calculations to engage and educate security leadership on alert quality.
• Reevaluate your analyst actions and training needs based on their rates.  
• Create feedback loops for the analysts to notify security engineering of ongoing problems. 
• Include these numbers in ongoing discussions with your vendors, specifically to improve the ongoing efficacy of their platforms. Seek to share these calculations with others in their customer community.