Embodied Multimodal Interaction with a Portable Mixed Reality-based Digital Tower

A major topic of air traffic research in the past decade has been of digital towers (DTs) that replace physical air traffic control towers by reproducing live camera feeds of out-of-tower (OOT) views using massive immersive spatial displays. With the recent pandemic-induced decline in air traffic, the quest to introduce cost-effective alternative solutions for DTs seems especially urgent. Recent developments in mixed reality (MR)-based visualizations have allowed complex spatial data to be better represented in a stereoscopic virtual space. Moreover, the immersion that MR offers is highly customizable, allowing controllers to maintain their view of the environment along with the OOT view, similar to a physical tower. Additionally, the increasingly robust technical capabilities of self-contained MR-based Head-Mounted Devices (HMDs) (such as Microsoft HoloLens 2) gives rise to the possibility of designing a cohesive and self-contained DT system on such a platform, making the DT system cost-effective, portable and interactive. In this paper, we have developed a simulated Digital Tower environment in mixed reality using Microsoft Hololens 2, with intuitive embodied and multimodal interactive capabilities. The system incorporates several visual features from real-world air traffic control environment and visualizes them in mixed reality with added interactive capabilities for aircrafts, gates and runways. We then assess the preliminary system with experienced air traffic controllers and our findings show that such a mixed reality representations of air traffic control is accepted by controllers as a useful way to visualize information on the OOT view. We also find that the visual and interactive affordances provided by the system shows promising potential to augment, and possibly replace present digital tower set-ups as a self-contained, interactive and portable DT system.