Charting New Territories in Immersive Experience Quality Engineering:
The 5 Vectors of Immersive Experiences
Unpublished white paper, Amdocs (2023)
The market for immersive experiences
Many companies have attempted to get ahead of the competition by exploring the possibilities of immersive experiences, and the market has seen both gains and failures. The release of Apple Vision Pro marks an interesting inflection point: it’s an immersive experience product that’s prohibitively expensive to most people; a promising novelty that represents an early stage in the lifecycle of something new. When immersive experience products become accessible to a critical mass of consumers, perhaps in the next couple of years, we are likely to see a rapidly growing market trend in this area. The quality engineering industry needs to be ready for this potential inflection point and sudden acceleration in immersive tech adoption.
At present, metaverse platforms aren’t capturing the interconnected universe of our imagination: products are disconnected from each other. However, as a company, Apple runs its ecosystems on principles of interoperability and complimentary services between products. Its attempt to create a mechanism for engaging with immersive experiences may be the harbinger of an alternative universe of digital tech.
The business case for immersive tech
From a business perspective, it can be difficult to justify investment in immersive experiences at this early stage of the market. While the metaverse concept itself may be easy to grasp, many organizations struggle to link these experiences to tangible business outcomes like return on investment (ROI) or customer satisfaction. When organizations roll out immersive tech, the key challenge will be ROI.
This is difficult for a company to deliver reliably when exploring a pioneering technology for the first time. Risk-averse companies will not be satisfied that they are able to deliver return on investment in an environment without sufficient reference points about what success might look like. However, forward-thinking organizations will invest in innovation for its own sake and may find opportunities to derive significant business value as an early presence in this space.
Currently, the most well-established applications of immersive tech are found in the video game industry, where games have been quick to adopt VR and AR to create more engaging experiences for players. However, some innovative companies in other sectors are starting to explore the potential of immersive technologies for their own business needs.
For example, telecommunications providers are experimenting with augmented reality to enhance customer support, using interactive visual overlays to guide users through tasks like setting up home Wi-Fi networks or troubleshooting device issues. In the financial services industry, institutions are investigating how virtual reality could be used to create immersive trading environments or to simulate complex financial scenarios for training purposes.
"If you look at the population in this region, the majority of people are completely on digital platforms, either they are using a mobile or tablet in whatever way they stay connected. Even if you look at the older population, you can see that many of the users are actually spending their time on social media - we wanted the crowd to be utilizing their time in a very productive way by attracting them into the new gaming and meta-world."
– Director of Quality Engineering, UAE
Despite these emerging use cases, widespread adoption of immersive tech faces significant technological hurdles. The current cost of immersive devices and infrastructure is prohibitively high for many consumers and businesses, limiting the potential user base. Advancements in hardware and software are needed to make immersive experiences more accessible, affordable, and user-friendly.
Once immersive tech has built up an audience and proved that it can offer a strong ROI, it will become part of the tapestry of how we create digital experiences, a medium or approach to be considered like any other. But to reach that point, vanguards in the industry will need to overcome technical and economic barriers, while demonstrating clear business value to drive broader investment and adoption.
An emerging market
Our survey findings that one in ten businesses (11%) has already begun to transform their enterprise using immersive technologies. By 2030, half expect to have laid the foundation for immersive technology investment: training up quality engineering teams (49%), preparing customer support services (48%), and starting research and development (46%). A third (35%) expect to complete these stages, launch their immersive tech products and services, and begin to see measurable outcomes. Only a small portion, one in five (20%) expect to pivot their business strategies around immersive tech.
The reason why these adoption figures seem low is due to the extensive digital transformation required to start deploying immersive technology. For instance, if companies need to use immersive technology for the Metaverse, virtual reality, or augmented reality, they will require a cloud-hosted network. This isn't a common setup yet, and most companies will eventually need to transition through 5G and increase their network bandwidth. They'll need to invest in infrastructure before moving to the cloud, where they can explore Metaverse environments like sandboxes. This extensive overhaul is all part of a broader digital transformation.
The industry understands that immersive tech will have far-reaching applications across business processes. Our survey QE experts are particularly intrigued by the transformative potential of immersive technologies in communications both with customers and between employees. Immersive tech offers a more dynamic experience than passive social media consumption. If businesses get it right, QE experts predict that immersive experiences will improve customer loyalty (83% agree), customer satisfaction (82%), and improve customer retention rates (81%).
The five dimensions of immersive quality engineering
Immersive technologies, such as virtual reality (VR) and augmented reality (AR), are rapidly evolving and becoming increasingly popular among businesses seeking to enhance customer engagement and create innovative offerings. However, the unique nature of immersive experiences presents new challenges for quality assurance professionals. Traditional testing methods may not adequately address the complex and multifaceted aspects of immersive technologies. This is not just a matter of developing new techniques to fill a capability gap - it requires a complete paradigm shift.
To bridge this gap, Ross. A McIntyre developed a novel five-dimensional framework specifically for evaluating the quality of immersive experiences. This framework goes beyond the scope of conventional testing approaches by considering five key dimensions: interaction, biometrics, technology, environment, and storytelling element. By examining each of these dimensions in depth, testers can gain a more comprehensive understanding of how users interact with, perceive, and respond to immersive experiences.
The framework, developed by Ross A. McIntyre, acknowledges the interconnectedness of these five dimensions and emphasizes the importance of assessing them holistically. For instance, the effectiveness of an immersive experience's storytelling element may be heavily influenced by the intuitiveness of its interaction mechanics and the consistency of its technology performance across different devices. Similarly, biometric data can provide valuable insights into users' emotional responses, which may be impacted by factors such as the comfort and safety of the physical environment that they are in e.g. at home in the living room, or in the office)
Interaction
Interaction describes the mechanics of how users behave in an immersive experience. For example, based on our experience of digital environments, we all know how to enlarge something on a screen by pinching our fingers and drawing them apart. But what is the vocabulary of interaction in immersive technologies? While current interaction patterns may rely on familiar, skeuomorphic representations such as dials and knobs, the goal for immersive experiences should be to go beyond merely replicating real-world interactions in a 3D environment. Designers should strive to develop new, more efficient interaction vocabularies that are uniquely suited to immersive contexts.
To achieve usability and audience growth, it's crucial to establish a common language of interaction that can be applied consistently across immersive technologies. Interactions should be intuitive, with information disclosed progressively to avoid cognitive overload. As innovative solutions are discovered and refined, they become part of a new interactive vernacular, balancing ease of use for digital natives with simplicity for a broader audience.
This dimension of quality engineering is the most fully explored of the five, as companies are already applying similar testing principles to digital environments, such as the quality of the UI/UX of an ecommerce website, like Amazon. However, immersive experiences demand a fresh approach that prioritizes efficiency and innovation over mere approximation of real-world behaviors. In this new vocabulary, it will be important to maintain a balance between intuitive UI/UX for digital natives and simplicity for a real world examples. The vocabulary of interaction within immersive experiences will be shaped by pioneers in this space, with contributions from across the technological landscape and nuances drawn from user behavior.
Quality engineering for immersive tech needs to move beyond just testing capabilities to an experience-led approach. The crucial balance is between validating core functionality ("Is it working?" "Can I use it?") and evaluating the seamlessness of the complete user journey. Quality Engineers will need to carefully weigh and prioritize these factors - ensuring it works properly while also making the overall experience intuitive and accessible. Navigating this balance allows them to forge an interactive vocabulary that resonates with both digital natives and those new to immersive environments.
Biometrics
Biometric data is provided by a user's biological responses when using immersive experience technology. Some data may be directly monitored by the platform, such as facial scans used to create avatars for virtual representation. This same technology could be applied to understanding micro-expressions and monitoring eye focus and dilation, allowing companies to make data-driven updates or launch products based on users' biological responses.
Biometric insights can also be gathered from other products within a user's ecosystem. For example, an Apple user might have an Apple Vision Pro headset, an Apple Watch, and an iPhone, each capable of collecting different types of biometric data. A smartwatch can monitor heart rate, which, if cross-referenced with data from the headset, could provide valuable insights into the user's emotional state at specific points in an immersive experience. By aggregating data from multiple devices, companies can build a nearly comprehensive profile of a user's physiological reactions.
As biometric testing evolves, it may expand to incorporate even deeper data points, such as brainwave activity, to gauge user responses with greater precision. While biometric testing is currently a niche aspect of quality engineering, there is significant potential for growth and refinement. Frameworks must be developed to enable testers to interpret biometric data more accurately, accounting for contextual factors that could influence readings. The ability to gather data from a range of products, including older models, allows for extensive biometric testing and a wealth of information that can be used to improve and optimize immersive experiences.
Technology
Technology, the third dimension of quality engineering for immersive experiences, focuses on achieving consistency across different types of immersive technologies. For example, users interacting with an experience through an Apple Vision Pro should have a comparable experience to those using a Meta Quest. Just as customer experience (CX) testing in the digital space aims to ensure consistency across all channels of a digital ecosystem, the same approach should be applied to immersive experiences.
To ensure a high-quality, consistent experience for the majority of users, testing efforts should prioritize the most widely adopted immersive platforms. similar to how digital testing suites have evolved to focus on the most popular devices and operating systems.
As the immersive technology landscape continues to evolve, quality engineering frameworks will need to adapt accordingly. Regularly assessing the market penetration and user preferences for different immersive platforms will be crucial in determining which technologies to prioritize in testing and optimization efforts. By staying attuned to these trends, companies can ensure their immersive experiences remain accessible and consistent for the widest possible audience.
Environment
Environment plays a critical role in shaping a user's interaction with an immersive experience, particularly in the context of mixed reality or hybrid experiences where users view digital elements overlaid onto the real world. To ensure broad accessibility and relevance, immersive experiences must be designed with a wide range of environments in mind, taking into account factors such as industry-specific workplace conditions.
For example, telecommunications companies can leverage immersive tech by offering customers virtualized "slices" of the network cloud, eliminating physical routers. Customers access personalized metaverse sandboxes from home, with telcos providing the underlying platform. Within these virtual environments, users subscribe to immersive experiences like streaming Disney+, virtual shopping at Zara, attending concerts via Ticketmaster, and more. Telcos package the entire digital realm with diverse content partners, accessible through an at-home virtual reality experience. While physically at home, users seamlessly transition into tailored VR worlds through devices provided by the telco, representing a radically new, immersive customer experience.
While creating experiences for users in more predictable environments, such as a home office, is relatively straightforward, the true challenge lies in developing immersive technologies that can seamlessly integrate into a variety of real-world contexts. As immersive experiences become more integrated into our daily work and leisure interactions, it's important to think about a variety of environments, so that the product doesn't become irrelevant or inaccessible to a large portion of its users.
Storytelling
Storytelling represents the final frontier of quality engineering for immersive experiences, playing a crucial role in engaging users and driving adoption. It's important for businesses to consider why people are using their immersive digital product. This question will shape the story they tell users. A compelling story not only communicates the practical benefits of the experience, such as increased efficiency or effectiveness, but also creates an emotional hook that resonates with users on a deeper level. How does an immersive experience make something better—more effective, or efficient—and why is it preferable to the alternative?
To understand the importance of storytelling in an immersive experience, one can draw a comparison to video games. Even if the narrative revolves around a fantastical setting with knights and dragons, players readily immerse themselves in the experience because they understand their purpose and role within the game,. why they are engaging with it, and what they are doing. Similarly, immersive experiences must craft stories that are equally captivating, providing users with a clear and compelling reason to engage with the technology.
Crafting a powerful narrative for an immersive experience requires a multifaceted approach, incorporating elements such as narrative depth (the complexity and richness of the storytelling in an immersive experience), user agency (the degree of control and interaction the user has within the immersive environment's narrative, and emotional resonance (the ability of the immersive experience to evoke emotional responses and connections with the user.). The story should be engaging and meaningful even when users are distracted or multitasking, reflecting the realities of how people interact with technology in their daily lives. Moreover, a well-crafted narrative can help businesses justify the investment in immersive technologies by clearly demonstrating the value and impact of the experience. 81% of businesses believe that narrative and storytelling will be a crucial element in the workforce needed to support quality immersive experiences. Narrative depth and user choice are integral parts of story design, and it's important that immersive tech is enjoyable even when users are distracted or multitasking.
At present, many of the immersive experiences available on the market may lack the necessary narrative components—the beginning, middle, and end—that define a truly convincing story. The right story, of course, will also answer questions that businesses might have about why they should fund an immersive experience. As the technology advances and more users adopt immersive experiences, the onus will be on creators to develop narratives that not only showcase the capabilities of the medium but also provide a compelling alternative to real-world interactions. The key to success lies in the ability to tell a story that captivates users and keeps them returning to the immersive world. As a critical mass of users begin to engage with these new digital worlds, that story may emerge in the telling.
To meet this challenge, businesses may need to develop new roles, such as "narrative designers," who specialize in crafting immersive stories that seamlessly blend technology, user experience, and emotional engagement.
In this new ecosystem, quality engineering must evolve beyond merely certifying functional requirements or technical elements. A comprehensive approach is needed that also accounts for certifying the narrative itself. From a quality perspective, it is crucial to assess whether the intended narrative truly compels users to engage with the experience as envisioned.
This necessitates a novel framework where quality engineers collaborate closely with narrative designers - an entirely new role and discipline. The test architects or pilots must have in-depth discussions with these narrative experts to understand and validate the storytelling objectives.
Certifying functionality, environment, and overall experience is no longer sufficient; quality processes must now also certify that the core narrative resonates with users and achieves its intended purpose within the immersive world. This represents a paradigm shift in how we approach quality assurance for emerging technologies.
A new model of testing
In this five-dimensional framework, everything is interlinked (for example, biometrics and environment) with storytelling providing a key axis running through every dimension. Within every dimension of the framework, there will be many components to be tested. This new framework will call for new ways of examining and evaluating results, with traditional KPIs such as user satisfaction surveys and active user counts falling short of the complex metrics involved in immersive environments. A new kind of testing environment is called for, with a fundamental shift away from centers of excellence to centers of enablement. These testing structures will continue to evolve alongside this emerging technology.
"Businesses are investing in UI/UX designers, who specialize in immersive technology to ensure that experiences are virtually stunning, easy to navigate and provide a sense of immersion. So, by privatizing UI/UX design companies into capital users, like I said earlier they make them feel connected to the virtual world and ultimately deliver an overall experience and immersive environment." (Software Development Engineer, US)
A user-centric approach
As businesses wonder how they’ll generate sales from immersive experiences, it’s still a good idea to apply the logic they’ve used to engage consumers with digital experiences over the past two decades. It’s essential for companies to take a user-centric approach to immersive experiences, investigating what users want to be able to do and why this product is uniquely suited to their requirements. At the moment, many are slow to adopt this approach, but eventually it seems likely that more will recognize that it is crucial for growth.
The importance of accessible design.
Our survey shows that organizations see the most important elements of an immersive experience as performance (79%), accuracy (79%), consistency (75%), and accessibility (75%). In recent years, it’s clear that businesses have increased their recognition of the importance of accessibility in product design. Companies are aware that media commentary is likely to ‘call out’ and critique immersive experiences that fail to address the needs of their entire audience. One key challenge is that while organizations typically rely on bodies like W3C for accessibility standards, there are no new guidelines specific to immersive experiences.
While the immersive technology space has seen failures like the ill-fated initial metaverse push due to technical limitations, poor user experiences, and prohibitive costs, we do not appear to be in a full "winter" period yet. The recent launch of Apple's Vision Pro suggests the industry could be on the cusp of broader adoption if the technology proves accessible, affordable, and delivers a compelling experience. However, if Vision Pro or other emerging immersive offerings fail to resonate with consumers, investment interest could once again temporarily chill, mirroring previous AI investment winters in the 1970s-80s and 1990s.
With immersive tech likely here to stay and poised for significant growth over the next decade, QE engineers should start preparing now as the technology becomes more affordable and accessible to a critical mass of consumers. Companies that deliver quality immersive experiences can greatly enhance their reputation and market position. The window for immersive success is open, and QE's role in validating quality, accessibility and seamless user journeys will be crucial for driving sustained interest from consumers and investors alike.
While internally, organizations seek to reduce bias with a well-balanced workforce, immersive tech will have to prove a company’s accessibility to potential audiences. However, this should not dampen innovation or an appetite for risk: companies that make well-meaning missteps are likely to be recognized for doing the best they can at an early stage of technological development.
Recommendations
Adopt the Five-Dimensional Framework: Businesses should incorporate the proposed five-dimensional framework (interaction, biometrics, technology, environment, and storytelling) into their quality engineering processes for immersive experiences. This holistic approach will ensure that all critical aspects of immersive experiences are thoroughly tested and optimized.
Prioritize User-Centric Design: Adopt a user-centric approach by investigating what users want to achieve with immersive experiences and why this technology is uniquely suited to their needs. Understanding user preferences and behaviors will be crucial for creating compelling and successful immersive products.
Emphasize Accessibility: Place a strong emphasis on accessible design from the outset. As immersive experiences become more prevalent, companies that fail to address the needs of their entire audience risk facing criticism and potentially losing market share. Develop processes to ensure accessibility is a core consideration throughout the product development lifecycle.
Invest in Biometric Testing: Explore the potential of biometric testing to gain insights into users' physiological and emotional responses to immersive experiences. This data can be invaluable for optimizing the user experience and driving product improvements.
Develop Storytelling Expertise: Recognizing the importance of compelling narratives in immersive experiences, businesses should consider creating roles such as "narrative designers" to craft immersive stories that seamlessly blend technology, user experience, and emotional engagement.
Conclusion
The emergence of immersive technologies presents both exciting opportunities and significant challenges for businesses. As this report highlights, traditional quality engineering approaches may not be sufficient for ensuring the success of immersive experiences. To navigate this new frontier, companies must adopt a multi-dimensional approach that accounts for the unique aspects of immersive technologies, such as interaction mechanics, biometric data, technological consistency, environmental factors, and compelling storytelling.
By embracing the proposed five-dimensional framework and prioritizing user-centric design, accessibility, biometric testing, and narrative expertise, businesses can position themselves as leaders in the immersive experience market. However, this journey will require a willingness to innovate, take calculated risks, and learn from well-meaning missteps along the way.
Ultimately, businesses that can successfully integrate immersive technologies into their offerings will not only enhance their reputation but also unlock new avenues for customer engagement, operational efficiency, and competitive advantage. As the report emphasizes, the quality engineering industry must be prepared to support this potential inflection point and the rapidly evolving landscape of immersive tech adoption.
Introduction
The next frontier of quality engineering is immersive experiences. These require businesses to take a new approach to gauging quality, using tools and approaches that they may not have used to test digital experiences in the past. Over the next five years, visual and audio quality will advance—but these components are just a small part of the experience of immersive tech.
At present, quality engineering strategies for testing immersive tech demonstrate some uncertainty about the best methodologies to use. In this chapter we introduce a five-dimensional framework to offer companies guidance on which aspects of quality they should consider when testing immersive experiences. We propose that the five vectors of an immersive experience quality framework are interaction, biometrics, technology, environment, and storytelling. As immersive tech increases in sophistication and innovation, this quality framework will evolve—but it offers a starting point when entering a pioneering experiential arena.