In the maturity process of any organization, at some point metrics enter the equation. For many, this is a cringe-worthy event that ranks up at the top of the worst job-defining moments.
It usually starts innocently.
Someone asks a question. It could be a simple question, such as “How much money are we spending and why?” or more pointedly “We are spending X dollars on intelligence services — exactly what return are we seeing?”
That starts the ball rolling.
Next is the invariable shuffle and dance to collect data—information that isn’t readily available, so it becomes a task on everyone’s list of items to do. Given the paucity of information at hand, really smart people with high analytical skills somehow seem to be completely dumbfounded on what to measure.
A bright idea is introduced to count whatever is at hand and so starts a very ugly cycle of counting widgets when wins are really what you are interested in.
A Clarification on Cyber Intel Metrics
To clear the air, metrics are incredibly important. For the technically-minded: it’s how you get more positions, more tools, better assets and frankly, better treatment. For the business-minded: metrics are justification, manpower planning, informed decision-making, asset allocation and proper key performance indicators.
In short, and for everyone: you need them for success.
Measuring the right ‘thing’ and in the ‘right way’ are the important parts of the process. You are measuring the end game. I’ll state it slightly differently for effect: it’s the outcome and effect of your intelligence work that matters here.
How many indicators you look at is interesting, but not relevant to figuring out how effective and useful you are. Measuring news events and how many vulnerabilities come out here, how many malware are written on there — all interesting, but not pertinent to outcome and effect. Attempting to calculate how much of anything can be a trap.
Items to Consider:
- Incidents prevented by providing controls, measures and mitigations.
- Decrease in incidents when compared to historical incidents.
- Risks, events and “issues” averted by intelligence-informed decision making
The first is a measure of items that intelligence has provided to security departments to implement a control to stop an event, a measure to handle an event if/when it happens or a mitigation to reduce the impact or outcome.
Metrics Tracking Examples
Example: An intelligence analysis cell informs the appropriate security department of a threat and equally provides the control, measure or mitigation to handle it.
Note: This is not helping make a detection for a threat but providing an (remember, outcomes are the measure here) to handle it. It’s one of those ‘truer’ measures of intelligence output that measure whether intelligence is assisting leaders in making informed decisions.
The impact of those controls, measures and mitigations are your next metric. How much did this action contribute to prevention, mitigation and control of incidents?
Example: Intelligence determines that a breach has occurred by research into external data. Providing the information and understanding of when the breach occurred, how it may/did happen and other details allows leaders to get in front of the problem. The breach is announced in a controlled manner, providing a chance to put a strong PR presence on the issue and maintain or even increase public opinion about “how well you handled” the issue
Finally, and admittedly harder to measure, is what you averted. What was contained before it really began or was flat-out deterred/prevented from occurring.
Example: Intelligence surveys and assesses public information exposure. Informing and educating the worst offenders of this problem can be mapped to phishing reduction. How? By analyzing the easy access of data you publish publicly to the phishing targets observed. It won’t be one-for-one matching, since information loss occurs via several avenues.
Example: Intelligence investigates third-party breaches to understand what corporate credentials are lost and their influence of attack vectors employed against the company. Beyond the simplistic application of determining password re-use is the more analytical world of determine the likelihood of targeting, be it phishing, account takeover, service compromising and many other threats linked to credential loss. Identifying and taking steps to secure those identities; or, possibly even turning them to a honey-pot to determine threats demonstrate this metric.
Example: Playing a part in the travel program of your company, intelligence determines the greatest threats to travelers–i.e. preventing information loss, credential loss, and other elements. Think controls like dedicated laptops that are wiped after each journey; dedicated SIM cards that are loaded with preconfigured information that provides only the basic necessities while introducing honey files to search for threat activity. Add on risk evaluations to prevent travel to dangerous areas, situations and so on.
The goal of defining quality Intelligence metrics is to constantly ask how it impacts the end game and the decision-making process.
The final benefit is not being satisfied and continually refining that measuring process for more comprehensive and more accurate results.