Mixed Reality App Design: Best Practices 1

(This is part 1 of the 2-part series about designing an enterprise app for Mixed Reality such as Microsoft HoloLens. Part 2 can be found here.)

Microsoft HoloLens, released March 2016, is still the world’s leading Mixed Reality platform. The freshly released Magic Leap One: Creator Edition will likely hold this honour until HoloLens 2 is released in 2019. HoloLens is geared towards custom enterprise solutions, which makes sense given the nascent phase of the Mixed Reality ecosystem.

Since most great design advances in the last few years have been made behind closed doors, the publicly available app that illustrates the best design principles for HoloLens by far the best is Fragments, a launch title from 2016 by the French game development studio Asobo. This article focuses on design lessons from Fragments for developing an enterprise app on HoloLens.

The characters in Fragments all show up in your room. First time this happens, it quickly hammers home the transformative power of Mixed Reality.

Fragments is a high-tech crime thriller where you play a detective who can walk in people’s memories. Passing from one memory to the next, you explore items in each space (which are shown in the context of your room where you are wearing the HoloLens) and pick up clues. Each solved clue lets you filter out potential locations on the map until you get down to one location where the suspect has been, thus moving the plot forward. As the story progresses, you pick up new capabilities for searching the environment including Ultra Violet, Infra Red, X-Ray, Audio and Heat sensing, and the spaces themselves get more complex and fun.

Fragments experience is several hours long and is worth your while not only because of the entertainment value, but because it offers many lessons on how to design a great Mixed Reality experience, whether for entertainment or for a business use case. Let’s explore these lessons in detail.

1. Talking is the best interface

Voice commands for every action. Hand gestures as back-up.

A detective needs to switch between tools, navigate menus, open and close things. Each such action can be performed by a voice command. The exact phrases to say are indicated by tooltips. HoloLens understands phrases said in different accents fast and with great accuracy. Overall, voice commands are easy to use and work very well.

Despite voice being the superior method, there must also be a way to perform each action by a gesture. Voice simply is not usable in all environments: when it is too noisy, when one must be quiet, when there is an issue with audio input.

Additionally, for spatial activities such as placing an object on a surface or moving it in three-dimensional space, using hand-gestures is the superior method. Yet holding the hand in front of one’s eyes and constantly “pinching” gets tiring quickly, so if your enterprise app requires a lot of interaction, consider the fatigue of the user.

Lastly, whenever one has the choice of naming something (in Fragments it would be the character, or the environment scanned by the app), offer a few sensible naming options and make the user pick one. Actually this applies to any open-ended questions; do not make a person type on HoloLens if at all possible.

Text and voice cues when “stuck”

When no progress has been detected for a while, a well designed app suggests what to do. These cues can be very simple tooltips such as “look here” displayed on the user’s gaze reticle, which also includes a pointer to the desired direction. This method is used extensively in Fragments, and it works great because it overcomes the two potential issues of identifying actionable phrases and the location where the user should be facing.

If user does not making any progress for a while, Fragments kindly starts giving out pointers on what to do next.

Another delivery method for cues is voice-overs. They are more expensive to produce than text and in case of any issues run into a greater risk of being repetitive and annoying. However, if used wisely and sparingly like in Fragments, they can be an elegant solution for nudging the user along. One must always remember though that the user might not hear voice-overs, and have a fallback plan.

2. Design around tech limitations

Render objects only at an ideal viewing distance

Fragments is doing several clever things for the user not to feel like they are looking into the “memory world” through a small window of HoloLens (which they are). Firstly, objects far from the user are not rendered. Only as the user gets close to the perfect viewing distance, the items in the world assemble themselves from “fragments” become visible. This way it is rare for the user to see objects in the background cut off by the “field of view window”. It is an ingenious trick and rendering technique that enterprise app developers could also make use of more.

Items and even characters get composed of “fragments” as the user gets to an optimal distance, thus making sure everything looks the best it can through the HoloLens headset.

On the other hand, Fragments makes sure its users do not end up seeing objects too close either, which again would result in them being cut off by the “field of view window”. As the user interacts with smaller objects, they “pop” next to one’s head in optimal viewing distance, where they can be inspected in more detail.

Room scan as part of the experience

If Fragments is launched in a new space for the first time, the app requires to scan the room. Scanning a space in Fragments is a playful experience where the user is encouraged to look in different directions and behind objects, and is provided visual rewards during and after the completion of the scanning. Your enterprise app can make use of the same principles, and even rationalize the scanning by some sort of in-app fiction if it makes sense. If the space is large, objects complex, and plenty of device movement is expected, a “scanning experience” might be the way to go. However, if these do not hold true, a Mixed Reality app could be developed that does not require any prior room scanning at all.

Scanning the room before the Fragments experience is pretty fun on its own.

For example, if the function of the app is to display a 3D model on a table, HoloLens will be able to scan the table on the fly, and do a decent job of tracking the position of the model on the table as the user moves around it. HoloLens will automatically keep scanning more of the room in the background (the more accurate information of the space it has, the better it is able to keep the content fixed in 3D space), so the tracking quality will become more accurate over the usage time of the app.

(This is the end of part 1. The final part 2 of the series can be found here.)

Leave a Reply

Your email address will not be published. Required fields are marked *

Social media & sharing icons powered by UltimatelySocial