Spotlight
NeoVision AR Windows That See What You See
Watch video of NeoVision AR.
Imagine standing in front of an aquarium window. A bright blue tang, the species behind Dory in Finding Nemo, swims by. Instead of fishing for information on your phone or reading static signage, you simply look at it, and the glass comes alive with facts about the fish. A simple hand gesture then reveals more details such as ecological facts and behavioral insights, turning an ordinary window into an interactive guide that responds naturally to human curiosity.
This is the experience enabled by NeoVision AR, an AI-powered transparent display that combines gaze detection, gesture control, and conversational interfaces.
NeoVision AR displays real-time marine life information aligned with the user’s gaze and gestures in an aquarium setting.
No Headsets, No Touch—Just Gaze and Gesture
Unlike conventional augmented reality systems that rely on headsets or handheld devices, NeoVision AR takes a different approach: it embeds intelligence directly into transparent displays.
The system combines gaze tracking, gesture recognition, computer vision, and conversational AI to understand what users are observing in the real world. By identifying viewing targets and spatial positions in real time, NeoVision AR then overlays relevant digital content directly onto the transparent display.
This multimodal AI architecture integrates scene understanding, object recognition, motion tracking, and natural voice interaction into a single interactive platform. Users can explore information naturally through eye focus, gestures, or spoken questions without touching a screen. AI avatars and voice interaction further enrich this experience, enabling users to ask questions or explore topics naturally.
Born From a Need for Contactless Experiences
The idea behind NeoVision AR emerged during the COVID-19 pandemic, when the world urgently needed safer, touch-free ways to interact with public information.
Researchers at ITRI began developing a transparent display system that could deliver immersive experiences without requiring physical contact. But building a system that understands real-world scenes in real time proved challenging, especially in complex environments like aquariums.
Water refraction, varying light conditions, and moving animals can easily confuse visual recognition systems. To overcome these, researchers trained AI perception models using tens of thousands of underwater images while optimizing camera placement, viewing angles, and multi-camera tracking strategies.
“Recognizing marine life through glass and water is significantly more challenging than standard object recognition scenarios,” said Chin-Ming Hao, Industrialization Director at ITRI’s Electronic and Optoelectronic System Research Laboratories. “NeoVision AR was trained to overcome those challenges and can now identify aquatic species and track their movement with up to 98% accuracy.”
The NeoVision AR research team poses with the silver award it received at the 2026 Edison Awards.
Turning Windows Into Smart Tourism Guides
NeoVision AR has already been deployed at the National Museum of Marine Science and Technology in Taiwan where it transforms aquarium viewing into an interactive educational experience. Beyond aquariums, the technology is also being adapted for tourism transportation, including ferries, sightseeing buses, and smart mobility applications. Vehicle windows become AI-enhanced guidance interfaces that recognize landmarks, scenery, and nearby points of interest in real time. Travelers no longer need to search for information on their phones or wonder, “What mountain is that?”, as they learn about their surroundings while experiencing them.
A Greener Way to Share Knowledge
By delivering information digitally and on demand, NeoVision AR reduces the need for printed signs, brochures, and maps. Information is refreshed digitally rather than through physical display changes, cutting waste while keeping content current.
The technology has already served millions of visitor interactions across museums, aquariums, and transportation settings. Beyond guiding tourists, it is opening new possibilities in education, cultural exhibitions, healthcare simulation, and smart public infrastructure, turning everyday windows into interactive portals for discovery.