Skip to main content

Explainable Metamodels for ATM Performance Assessment

Paper ID

SIDs-2022-075

Conference

SESAR Innovation Days

Year

2022

Theme

Modelling and Explainability

Project Name

SESAR 2020 ER4 project NOSTROMO

Keywords:

Air Traffic Management Simulation Modeling, Model Explainability, SHAP values, Simulation Metamodeling, xgboost

Authors

Christoffer Riis, Francisco Antunes, Tatjana Bolic, Gérald Gurtner, Francisco Pereira and Carlos Azevedo

DOI

Project Number

892517

Abstract

Fast-time simulation constitutes a well-known and long-established technique within the Air Traffic Management (ATM) community. However, it is often the case that simulation input and output spaces are underutilized, limiting the full understandability, transparency, and interpretability of the obtained results. In this paper, we propose a methodology that combines simulation metamodeling and SHapley Additive exPlanations (SHAP) values, aimed at uncovering the intricate hidden relationships among the input and output variables of a simulated ATM system in a rather practical way. Whereas metamodeling provides explicit functional approximations mimicking the behavior of the simulators, the SHAP-based analysis delivers a systematic framework for improving their explainability. We illustrate our approach using a state-of-the-art ATM simulator across two case studies in which two delay-centered performance metrics are analyzed. The results show that the proposed methodology can effectively make simulation and its results more explainable, facilitating the interpretation of the obtained emergent behavior, and additionally opening new opportunities towards novel performance assessment processes within the ATM research field.