An engineer visualizes a mechanical hologram with Microsoft’s Hololens smart glass technology. Photo courtesy of Microsoft
Research, operation and future considerations for augmented reality technologies have been expanding throughout the global aviation industry over the past decade, though published exploration of A/R can be traced back as far as the 1990s. We highlight some of the ways in which different areas of the aviation industry are using and evaluating future use cases for augmented and virtual reality technologies.
According to Intel, the official definition for augmented reality is computer-superimposed enhancements to a user’s real-world environment, such as an oil pressure reading on a cockpit panel display. Virtual reality is defined as the creation of an environment by a computer that a person fully sees — usually accomplished with goggles or a head-mounted display system. For example, using virtual reality, the user can experience stepping into an airplane avionics bay or cabin.
Pratt & Whitney
Photo courtesy of Pratt & Whitney
Pratt & Whitney is no stranger to pushing new technologies forward. As far back as 2002, for example, Pratt & Whitney Canada announced an agreement with IBM and Dassault Systems to become the first company in the aerospace industry to develop engines using digital technology throughout the entire design and manufacturing process.
In April, the United Technologies Research Center unveiled a collaboration with Pratt & Whitney’s customer training division to invest in virtual reality engine maintenance training for airline mechanics. According to Bruce Hall, general manager of Pratt’s customer training division, the company is currently beta testing in classroom environments the use of headsets and hand sensor controls that would allow mechanics to virtually walk inside a GTF engine to examine parts and view a running engine in motion.
Photo courtesy of Aero Glass
Aero Glass has a headset that pilots can wear and view cockpit control information like altimeter readings, fuel pressure, heading and oil temperature within a display that sits in the glass portion of the headset. Aero Glass made headlines in October 2016 when Airbus BizLab selected their technology as one to help become transformed into a business proposition. The head-worn display concept has also received funding from the European Union’s Horizon 2020 research and innovation program. Check out its 2014 YouTube video below detailing the concept.
SkyLights headset. Photo courtesy of SkyLights
Air France began testing Aug. 1 what it calls an “immersive entertainment system” with the use of virtual reality headsets that passengers can wear to view 3D and 2D films or television series.
The virtual reality headset is a result of the French carrier’s partnership with SkyLights, an American-French startup company that was awarded ‘Les As De L’Innovation’ at the Paris Air Forum in July. Under the award, SkyLights will receive funding from Air France to expand development of its headsets.
Air France is testing the use of the headsets onboard its Airbus A340 flights between Paris-Charles de Gaulle and St. Martin. At the end of this test period, this new system could be rolled out on other flights in the months ahead, according to Air France.
Air New Zealand
Microsoft describes its Hololens headset as a “fully self-contained holographic computer” with an optical system that works with advanced sensors and a holographic processing unit (HPU). The HPU is a a TSMC-fabricated 28 nm coprocessor that has 24 Tensilica DSP cores. It has around 65 million logic gates, 8 MB of SRAM, and an additional layer of 1 GB of low-power DDR3 RAM. That RAM is separate to the 1 GB that's available for the Intel Atom Cherry Trail processor, and the HPU itself can handle around a trillion calculations per second.
Air New Zealand, in collaboration with IT service-provider Dimension Data, is beta-testing the use of HoloLens for its cabin crew. The airline envisions a future where flight attendants wearing a HoloLens headset can display passenger information on the headset such as flight details, time since last served and even the emotional state of the passenger. The video above gives an overview of what Air New Zealand wants to do with the technology.
Bell's FCX-001 concept helicopter at Heli Expo 2017.
At Heli Expo 2017, Bell Helicopter publicly unveiled for the first time its futuristic FCX-001 concept helicopter, with an airframe crafted from sustainable materials, a hybrid power system, an artificial intelligence co-pilot and morphing rotor blades that change to suit different flight conditions.
Bell’s description of the virtual cockpit feature notes that the company sees “pilots of the future controlling the aircraft with the aid of augmented reality and an artificial intelligence computer assistance system.” This would place future helicopter pilots in the role of safety and mission officer, while the computer assistance system flies with them.
Bell sees the virtual cockpit concept as a stepping stone to "fully autonomous un-piloted vertical-takeoff-and-landing air vehicles in the future."
Photo courtesy of Fountx
In March, TAE Aerospace received the Aerospace Australia Civil Industry Innovation award for its Fountx wearable technology designed to allow an on-site aviation technician to collaborate with product expert remotely.
Developed in partnership with Australian industrial research organization CSIRO, Fountx uses a real-time audio-visual system consisting of an operator headset and a station from which the expert relays guidance on how to complete specific tasks.
TAE’s development of the concept is spurred by the global nature of its operations. As the largest turbine-engine maintenance provider in Australia, its technicians are on call 24 hours per day, providing assistance to airline operators, private pilots and defense forces in New Zealand, Indonesia, Taiwan, Malaysia, Thailand, Sri Lanka, Nepal, Africa and North America. Its shared reality system technology runs on Microsoft’s Windows 10 operating system.
The technology could provide assistance, for example, to the Royal Malaysian Air Force and the Royal Thai Navy in repairing turbine engines, both of which are TAE clients. TAE was also recently selected to provide support for F-35 fighter engines operating in the Asia Pacific region.
In 2016, Airbus funded a flight test to evaluate its DS Electronics and Border Security SFERION situation awareness system that uses the operational SferiSense 500 LiDAR. The system was installed on the University of Iowa’s Operators Performance Laboratory (OPL) Mi-2 helicopter, together with a helmet-mounted display and an optical head tracker. The goal was to demonstrate and test the integrated system in the DVE-M flight trials under brownout conditions at Yuma Proving Ground.
C360 augmented reality video technology. Photo courtesy of C360 Technologies
Boeing announced Aug. 2 the latest investment by its HorizonX venture arm, which was established earlier this year with the purpose of investing in startups for disruptive new aviation technologies. The latest investment positions Boeing within a group of investors funding Pittsburgh-based C360 Technologies’ non-stitched immersive video in the world. Our live output for linear broadcast and OTT streaming platforms coupled with the industry's smallest pro-grade immersive camera is unique compared to others.
Boeing gave little details about future use of C360’s capabilities, but did state that the “potential aerospace applications” include “more capable autonomous systems and other advanced platforms.”
A commercial airplane cockpit as seen through HoloLens. Photo courtesy of Microsoft
Japan Airlines first trialed the use of Google Glass for their maintenance operations at its Honolulu station in 2014 with the aim to increase work efficiency. During the trial, maintenance staff could receive advice and instruction by audio during operations as well as photograph or video real-time information that could be shared with colleagues off site.
However, when Google removed Google Glass from the commercial marketplace, the Japanese carrier moved on and found a new use for Microsoft’s HoloLens headset last year. Japan is using the HoloLens virtual reality headset to train new engine mechanics and flight crew members. For example, in place of traditional printouts of cockpits or engine components, mechanics in training can walk inside of an actual virtual engine or cockpit and learn how to work on them.