Critiquing a Risk Analysis Method
In this module, your task is to thoroughly read and understand a risk analysis method developed by scientists at a national laboratory for use in managing terrorism and security risk.
- Read the report “Novel Threat Index using PRA.”
- Who are the authors, and what might be their point of view?
The authors of the research paper, Novel Threat-Index Using Probabilistic Risk Assessment and Human Reliability Analysis, were members of the Idaho National Engineering and Environmental Laboratory – Bechtel BWXT Idaho, LLC. They comprise of; Martin M. Plum, David I. Gertman, George A. Beitel, Jerry H. Phillips, James A. Vail, Ronald L. Boring, Patrick H. McCabe, Kyle S. Staples, Jeffrey C. joe, David H. van Haaften, Robert E. Polk, and Garrold L. Sommers.
The authors propose that it is possible to enhance decision making in detecting and responding to the risk of terrorist attacks, particularly on infrastructures such as dams by applying the Quantitative Threat-Risk Model (QTRIM). QTRIM is a comprehensive comprised of subset models that incorporate the use risk, physical, economic engineering and engineering behaviour variables to quantitively determine the risk of an attack of United States infrastructure by terrorists. The authors opine that prior research works in estimating the probability of terrorist attacks were majorly qualitative, hence one dimensional and inaccurate. QTRIM is cited as a novel model that should form the basis of further research in the model-oriented decision of the threat and risk posed by terrorist attacks on US assets (Martin M. et al, 2004, p. 4-11).
- What is the question that this method is trying to answer?
This method attempts to answer the question; How can the probability of a terrorist attack on US assets be detected and the possibility of enhancing decision making during the process. Another question that the method also tries to answer is; is it possible to quantitatively estimate the risk of terrorist attacks on US assets. Notably, previous research in this field has mainly focused on qualitative variables that may fail to give a complete perspective of risk assessment.
This method also aims to answer the question; if it is possible to develop a terrorist attack risk model that can compare and provide interrelationships between different facilities or assets.
- As a system, how does the method work? That is, what are the inputs, what are the outputs, and how do inputs generate outputs?
The QTRIM employs a subset of models that incorporate the use risk, physical, economic engineering and engineering behavior variables to determine the risk of attack on assets quantitively (Martin M. et al, 2004, p. 4). The model simulates the probability of a facility being targeted by terrorists by the use of a set of input and output variables. Normalized ranking parameters are used to differentiate the varying probability of one asset being attacked compared to another.
The model annotates investment to mean the input variables to the process, and return on investment as the output variables as below;
Investment resource inputs that comprise of success probability of an attack, people, the resources to carry out the attack and the schedule. These variables are multiplied to obtain an investment score (‘A’), whose feasibility is determined by how high it is. A higher investment score would translate to a better target opportunity (Martin M. et al, 2004, p. 5).
- Return on investment
The output variables were taken as the probable end objectives that terrorists intend to achieve when attacking a facility. These objectives include human deaths, economic loss, a decline of Western Powers’ presence in Islamic states, opportunities to leverage with like-minded terrorist organizations, inconveniences to the target Country or facility, and enhancement of Islamic presence for radical terrorists. Similar to the investment variables, the output variables are multiplied to obtain a return on investment score (‘B’) (Martin M. et al, 2004, p. 5).
Multiplication of the investment score and return on investment score yields the selection score, which determines the ranking of targets as probable hit targets by terrorist organizations.
- How do the authors justify the equations they use for this method?
The authors have majorly used assumptions and subjective probability to justify the equations used in the model. Some examples include;
- Determining the number of terrorists
To calculate the people factor x’, the authors warrant higher weighting smaller numbers of terrorists and vice versa, by postulating that large terrorist cells are easier to detect and hence thwarted compared to smaller cells which are more effective.
- Terrorist resources
The calculation of resource factor, y’, is justified by estimating the unlikelihood of terrorists using explosives weighing more than 1,000,000 pounds due to logistical and acquisition challenges.
- Terrorist Schedule
The authors have justified the schedule factor to have an inverse relationship with the time required for planning, mobilization and actual attack since a more prolonged period would likely result to the probability of detection or underlying assumptions to change.
- What assumptions does this method make?
This method makes several assumptions that include;
- Terrorists are rational and will focus on targets based on a probable return to investment.
- The model assumes a logical planner of terrorist activities
- The terrorist organization has informational access of targets, e.g. via online sources or using ground accomplices
- The terrorists are knowledgeable in balanced scorecard techniques
- Decision making in a terrorist organization is made in a limited process. This assumption means that only a few leaders make the decisions and the rest of the members follow cordially.
- From where is the method user supposed to get the data to apply this method?
- Interviews and procedures with human sources
- Online resources
- Assessment of target equipment and facilities
- Other research works
- What does the output from this method look like?
The output of the method is a quantitative perspective of the risk probability of a terrorist attack on various assets, with the key outputs being the level of risk, the risk value in monetary terms, frequency of potential attacks and associated fatalities if any. The output can also be summarized using a risk scale.
Below is an output example of the model;
Consequence categories for a model on asset X;
Low risk: < $ 500,000, or potential death of 2 persons
Frequency category of potential attack;
Medium: 0.0001 per year
- What factors call into question the credibility of the results from this method?
- Majority of the assumptions used in the study, especially in developing equations, have not been verified and may be overly subjective.
- In contrast to the assertion made by the authors that the model applies to different assets, the particular model is heavily reliant on specific input variables and is inclined towards the assessment of dams
- The method does not consider many dynamics of terrorist attacks and is modelled primarily on Al Qaida-type terrorists only leaving out many other possibilities of terrorist attacks
- The method is based on estimating future terrorist attack risks based on historical information. In some case, the future does not always evolve as past periods.
- The margin of describing risk as low, medium and high lack distinctive and may be considered more subjective than objective.
- Would you trust the outputs from this method? Why or why not? If your answer is “it depends,” under what circumstances are the outputs trustworthy and not trustworthy?
The authors have made a modest effort to develop a model that attempts to estimate the risk of terrorist attacks on various assets quantitatively. The outputs of this method are only trustworthy hypothetically to the extent that is accurate in line with given assumptions. For example, the QTRIM outputs in the report are precise only based on the assumptions that only Al Qaida-type terrorists are considered in the model (Martin M. et al, 2004, p. 16). Notwithstanding, the model would be heavily inaccurate in real-time mainly because it has not considered the many variables that would influence and suffice a terrorist attack on an asset. The report, however, provides a sound basis for further improvement to more accurately detect risk in terrorist attacks and consequent decision-making processes.
For this module, you are asked to consider the role of analytic confidence in risk and intelligence analysis. Your response to this module should be packaged as a single PDF file that addresses the questions below. Resources for this module include the following:
- PAPER: Appropriate Factors to Consider When Assessing Analytic Confidence in Intelligence Analysis
- PAPER: Intelligence Analysis and Judgmental Calibration GAME: “Credence.” (Links to an external site.)
- WEBSITE: Predictionbook.com (Links to an external site.) – You need to create an account to use this site.
- IMAGE: Types of Analysts
Answer the following questions based on your review of the above resources:
- What is analytic confidence and what role does analytic confidence play in risk analysis?
Analytic confidence is an assessment of an analyst’s confidence that what one has estimated or forecasted is accurate (Joshua, 2008, p. 6). Risk analysis usually involves consideration of dynamic factors such as complexity, reliability of sources and methodology of deducing information which would require analysts to provide their analytic confidence on their output to give particular weighting on the same. An analyst’s analytic confidence, especially in high risk matters, would determine the intensity and nature of policy steps undertaken to avert certain phenomena from happening.
- What factors are considered to be the appropriate ones for assessing analytic confidence?
- Expertise in the subject matter
Expertise in the subject matter may affect the analytic confidence of an individual both positively and negatively. There is a possibility of experts in a field to be overconfident on their capability to predict outcomes in their domain, compared to non-experts. However, experts are better higher analytic confidence in their core areas compared to non-experts (Joshua, 2008, p. 11).
The time to analyze and work on reports affects the analytic confidence of individuals. The analytic confidence of individuals working on tasks with time constraints is likely to be lower compared to jobs with adequate time. However, tasks that have no or unlimited time limitations may result in lower analytic confidence levels (Joshua, 2008, p. 11).
- Complexity of tasks
Difficult tasks lower the analytic confidence of individuals as compared to lighter tasks.
- Reliability and conflict of sources
When information sources provide collaborating information, the analytic confidence is usually higher while conflicting sources tend to lower the analytic confidence of individuals.
- What is judgmental calibration? For each of the analysts describe in the Types of Analysts image, label their analytic performance as calibrated, overconfident, underconfident, or otherwise. Explain why you assigned each label.
Judgmental calibration relates to the ability of an individual’s subjective estimates to correspond to correct estimates at a given level of probability (Steven, 2004, p. 1). In other terms, judgmental calibration defines the accuracy of an individual’s assumption.
Analyst A: Overconfident
Analyst B: Overconfident (No correlation)
Analyst C: Calibrated
Analyst D: Underconfident
- Play Credence a few times. Explain the rationale for such a game. Also, explain how the game works, how it provides you with feedback, etc. What do the graphs mean? Take a screen shot of the game.
The credence game aims to improve an individual’s calibration. The game is structured to have questions with two choices to choose. For each item, the player should indicate their percentage of credence levels (Andrew, 2020). The game uses a scoring rule to give the player points based on questions correctly answered that match credence levels. The points garnered determine if the player is overconfident, underconfident or well calibrated. The results may then be tabulated in graphs.
Figure 1: Screenshot of Credence Game
- Review the Predictionbook.com website. As a tool, what is this site supposed to do, and how does it work? Do you think it will work? Why or why not?
The Predictionbook.com website assists individuals to test their credence levels and obtain feedback on how accurate their estimates or projections were. Typically, a new user signs up into the website and creates forecasts or estimates on varied subjects and their confidence level of an event happening which the website stores. Once the subject event happens in real-time, the website assists in determining the credence level of the individual, e.g. if they were overconfident in their prediction (Prediction Book, 2020). By providing a platform for indicating forecasts and confidence levels, as well as helping participants to obtain the results of their predictions, the site is an important tool that can assist in judgmental calibration.
Andrew C. (2020). The Credence Calibration Game, by CFAR. https://acritch.com/credence-game/
Joshua J. P (2008). Appropriate Factors to Consider When Assessing Analytic Confidence in Intelligence Analysis
Martin M. et al. (2004). Novel Threat-Risk Index Using Probabilistic Risk Assessment and Human Reliability Analysis. Idaho National Engineering and Environmental Laboratory
Prediction Book. (2020). https://predictionbook.com/
Steven R. (2004). Intelligence Analysis and Judgmental Calibration. International Journal of Intelligence and CounterIntelligence