ASHiCS
ASHICS Automating the Search for Hazards in Complex Systems
Contents
About
Safety analysts are starting to worry that large complex systems are becoming too difficult to analyze when part of the system is changed or placed under stress. Traditional safety analysis techniques may miss safety hazards or (more likely) some of the circumstances that can cause them. To help analysts discover hazards in complex systems, ASHiCS has created a proof-of-concept tool that uses evolutionary search and fast-time air traffic control (ATC) simulation to uncover airspace hazards that might otherwise be missed using traditional manual safety analysis.
Methodology
In the ASHiCS process, simulations sit within what is termed a “search harness”. This describes the evolutionary computation software that “wraps” around the simulation, allowing the search to automatically start, configure, stop and select those simulation runs that are of interest to us. In our case, a simulation that results in a hazard or risk is of interest, and will therefore be judged to have higher “fitness”. The harness ranks the best individuals and creates mutated copies of these for the next generation to see if their fitness can be improved. This next generation of simulations is run, and again each is assessed for fitness. The process is repeated until the levels of evolved fitness in the population either reach a plateau (where no more improvement is likely or possible) or a sufficiently good simulation is found that allows us to stop the search.
Case study
As a specific case study, ASHiCS created a fast-time ATC simulation of an en-route sector containing multiple flight paths and aircraft types, and into each run of this simulation it was injected a serious incident (cabin pressure loss) that requires one aircraft to make an emergency descent. To create additional complexity and extra workload for the air traffic controller (ATCo), it was also introduced a storm moving across the sector. A screenshot of the resulting simulation is shown in Figure 1.
Figure 1 – Screenshot of the ASHiCS case study scenario in RAMS
Given that simulation, it was then used a near-neighbor random hill-climber to search for high-risk variants of that situation: a wide range of variants was run, the subset of variants that caused the most risk was selected, and then mutate the aircraft entry times to create a new set of situation variants that will hopefully have even greater risk. Figure 1 shows a typical progress of a search over time – individual runs vary widely in fitness (i.e. in the level of risk they exhibit), but the trend is for the highest-risk run so far (horizontal bars) to increase over time.
Figure 2 – Progress of a search over time – horizontal axis is number of simulation runs, vertical axis is fitness (risk) of each run The “search space” (the set of all possible simulation runs) is extremely large and cannot be exhaustively searched for the worst case; this is a problem for safety analysts who need a context to the search results so that they can determine event probabilities. The approach taken was to provide a local context for the search results – for each high-risk situation found, the space of situations that are very similar are explored. This cannot demonstrate that the worst case scenario has been found, but it can indicate the expected frequency of that result in its near neighborhood. This provides some insight to the nature of the solution space within the near neighborhood of the original result, in terms of the frequency of high risk scenarios and how those scenarios differ from the original. Figure 3 illustrates the second-stage search for local context by plotting 5000 samples of the near neighbourhood of the best scenario from a single first-stage search. The fitness score of the original is shown as a continuous horizontal line just below the fitness score of 2500 (vertical axis).
Figure 3 – Near neighbour sampling of original search result
Two things are interesting about this result. First, it can be seen that the search failed to find the worst case scenario. By extensively sampling the near neighbourhood, it was uncovered 3 variants who improved on the fitness score of the original scenario (these are the three dots above the horizontal line). However, what is also apparent is the original search result is at (or very close) to the worst cases, something which have been confirmed by careful comparison of the aircraft entry times involved in conflicts. It would appear that although some marginal improvement is possible, the search performed well in terms of finding the worst case scenario. Secondly, it can be seen that the vast majority of variants, even within this narrowly defined near neighbourhood of a high ranking scenario, do not come anywhere near the original fitness score. This suggests that there is a relatively narrow parameter band that generated the original high scoring scenario and its close variants. (From our analysis of the variant entry times, it appears that slight variants of the entry times of just 3 out of 20 aircraft are responsible for all the reported conflicts.)
Overall results
The overall ASHiCS process produces a set of high-risk variant situations, which can then be studied in depth. This study can start in the original simulation, and then progress to higher-fidelity models and complementary analysis approaches. The contribution of ASHiCS is to identify the situation types that that generate the worst cases; analysts and can then investigate how to prevent that configuration of inputs leading to a hazard in the air sector being modeled. The aim of ASHiCS was to develop a proof of concept approach to the automatic identification of hazards in complex systems. Our two-stage search process not only demonstrates the identification of hazardous scenarios, it helps analysts to understand the context in which these hazards occur and thus their place in the risk landscape of the whole system. With further work the approach would provide valuable practical benefits.
Reports
The following ASHICS project papers can be downloaded here:
Scenario Description Technical Report: This report lists the ASHICS project’s requirements and explains why the project selected ATM simulation software as its basis.
Baseline Scenario and Search Description Technical Report: Discussions relating to the choice of air traffic scenarios and types of hazards and risk models.
Risk Measures Technical Report: This report details the development of the search harness around the simulation software and the heuristic algorithms that guide the search.
References
[1] Clegg K, Alexander R. ASHiCS: Automating the Search for Hazards in Complex Systems. Proceedings of the 1st SESAR Innovation Days, Toulouse, France, 2011. [2] Clegg K, Alexander R. Searching air sectors for risk. Proceedings of the 2nd SESAR Innovation Days, Braunschweig, Germany, 2012. [3] Clegg K, Alexander R. Searching for Risk in Large Complex Spaces. Proceedings of EvoStar, Vienna, Austria, 2013. [4] Clegg K, Alexander R. The Discovery and Quantification of Risk in High Dimensional Search Spaces. Proceedings of the Genetic and Evolutionary Computation Conference (GECCO), Amsterdam, 2013
Research Team
Coordinator: University of York
