Competing on Analytics

Transcription

1 Competing on Analytics Probability explains the limitations of our knowledge of truth, it never denies it. - William Briggs, Uncertainty: The soul of modeling, probability & statistics

2 Aim: To convince you that we, as security leaders, must compete with the bad guys on analytics Q: Why is this important? A: We re losing and bode to lose more.. Q: What s the problem? A: Our concept of measurement* gives advantage to the bad guys and under performing vendors. Q: What s the solution? A: An integrated approach for measuring security risk. * We will use the term measurement and analytics interchangeably.

3 Why?: We re losing. There is nothing more deceptive than an obvious fact. Sherlock Holmes The Bascombe Valley Mystery

4 We re losing. James B. Comey, Director FBI, made the following statement before the Senate Committee on Homeland Security and Government Affairs on Nov 14, 2013: The diverse threats we face are increasingly cyber-based. Much of America s most sensitive data is stored on computers. We are losing data, money, and ideas through cyber intrusions. This threatens innovation and, as citizens, we are also increasingly vulnerable to losing our personal information. That is why we anticipate that in the future, resources devoted to cyber-based threats will equal or even eclipse the resources devoted to non-cyber based terrorist threats..[fbi]

5 We re losing The Global Information Security Workforce Study (GISWS) a survey conducted in 2015 of more than 14,000 security professionals, including 1,800 federal employees showed we are not just taking a beating, we are backpedaling: When we consider the amount of effort dedicated over the past two years to furthering the security readiness of federal systems and the nation s overall security posture, our hope was to see an obvious step forward. The data shows that, in fact, we have taken a step back. (ISC) 2 on the announcement of the GISWS, 20153

6 We re losing The founder and head of XL Catlin, the largest insurer in Lloyd s of London, said cybersecurity is the biggest, most systemic risk he has seen in his 42 years in insurance. Potential weaknesses in widely used software; interdependent network access between companies, vendors, and clients; and the possibility of large coordinated attacks can affect much more than even one big company like Anthem, Target, or Sony. XL Catlin believes it is possible that there could be a simultaneous impact on multiple major organizations affecting the entire economy. They feel that if there are multiple major claims in a short period of time, this is a bigger burden than insurers can realistically cover.

7 and will lose a lot more due to the exploding attack surface and its amplified uncertainty.. Internet users worldwide grew by a factor of 6 from 2001 to 2014 (half a billion to 3 billion). Since 2001 the number of websites alone has grown at a rate five times faster than the number of users a billion total by The emerging platform economy that competes on speed (devops) will exacerbate this in ways yet to be seen 4.9 billion connected things will be in use in 2015, up 30 percent from 2014, and will reach 25 billion by Gartner Breach will increase. This is due to the increase in connected systems, users, devices and uses per device with potential vulnerabilities, even if vulnerabilities per system or device do not increase.

8 The Problem: Quant education & application Success is a function of persistence and doggedness and the willingness to work hard for twenty-two minutes to make sense of something that most people would give up on after thirty seconds. Malcom Gladwell Outliers

9 The Problem: We need more quant exposure For the book, we conducted a survey of 171 Cybersecurity professionals. Strong opinions against quant are associated with poor stats understanding. Note: Those who performed the worst on the stats literacy quiz were more likely to overestimate their skills in statistics.

10 The Problem: We are not consistent (Responses: Agree, Disagree, No Opinion/Don t Know) Information security is too complex to model with probabilistic methods. Management and users won't understand the quantitative methods output. An expert using quantitative probabilistic methods will do better risk assessments then an expert using intuition alone. RESULTS: 80% of respondents had more pro than anti quantitative responses. Only 22% were consistently pro on quantitative and anti on softer scoring methods.

11 The Problem: Even the basics are misunderstood Assume that you have a portfolio of systems for which you have observed no security events in the past year that resulted in a monetary or productivity loss, which of the following statements is true? r a! r r r Answer Options If no events were observed, then we have no data about the likelihood of these events. The fact that no events were observed tells us something about the likelihood of these events. One year is not long enough time to gather enough observations to make an inference. Since some events may not have been observed, the lack of observed losses tells us nothing. There is insufficient information to answer the question. I don't know Response Percent 2.2% 37.0% 4.4% 31.9% 17.8% 6.7%

12 Solution: The current approach is useless Q: How many have used risk matrices similar to the one below, or have security management solutions (SIEM, VM, TM, AV etc) that use similar scales for prioritize risk? Tony Cox What s wrong with Risk Matrices investigates various mathematical consequences of ordinal scales on a matrix. they can be worse than useless, leading to worse-than-random decisions. Bickel et al. The Risk of Using Risk Matrices, Society of Petroleum Engineers, 2014 The burden of proof is squarely on the shoulders of those who would recommend the use of such methods to prove that these obvious inconsistencies do not impair decision making, much less improve it, as is often claimed.

13 The Problem: Communication is an illusion Budescu et. al. Psychological Science, 2009 on the use of verbal, qualitative scales for likelihoods: [Verbal terms] induce an illusion of communication People assume that everyone interprets the terms consistently and similarly, and fail to appreciate the variance in the interpretations of these words. The high level of potential miscommunication has been widely documented in many contexts. Richards Heuer, The Psychology of Intelligence Analysis, Center for the Study of Intelligence, CIA, % 20% 30% 40% 50% 60% 70% 80% 90% 23 NATO officers estimates of probabilities for events described using common terms used in communicating likelihoods in intelligence reports (e.g. War between X and Y is )

14 The Problem: We live on placebos The first principle is that you must not fool yourself, and you are the easiest person to fool. Richard P. Feynman Collecting more than a few data points on horses makes experts worse at estimating outcomes. (Tsai, Klayman, Hastie) Interaction with others only improves estimates up to a point, then they get worse. (Heath, Gonzalez) Collecting more data about investments makes people worse at investing. Collecting more data about students makes counselors worse at predicting student performance. (Andreassen) An experiment with a structured decision analysis method shows confidence increased whether decisions are improved or degraded. (Williams, Dennis, Stam, Aronson) In short, we should assume increased confidence from analysis is a placebo. Real benefits have to be measured.

15 Solution: The right concept of measurement For those who believe something to be immeasurable, the concept of measurement or rather the misconception of it is probably the most important obstacle to overcome. For all practical decision-making purposes, we need to treat measurement as observations that quantitatively reduce uncertainty. HTMA Cyber As far as the propositions of mathematics refer to reality, they are not certain; and as far as they are certain, they do not refer to reality. Albert Einstein Although this may seem a paradox, all exact science is based on the idea of approximation. If a man tells you he knows a thing exactly, then you can be safe in inferring that you are speaking to an inexact man. Bertrand Russell

16 Solution: An integrated security analytics approach

17 Solution: Basic Sparse Data Analytics: Monte Carlo Simulations

18 Solution: Advance Sparse Data Analytics: Monte Carlo Mitigation Simulations

19 Solution: Advance Sparse Data Analytics: Multi-Portfolio Monte Carlo Mitigation Simulations

20 Solution: Rolling packages in R!

21 Solutions: Security Metrics Survival Analysis

22 Solutions: Security Metrics