Saving Lives with Mixed Reality • Mathijs Sterrenburg

Mixed Reality - CPR, firefighting & medical training

Mixed Reality - CPR,
firefighting & medical training

Mixed Reality -
CPR, firefighting
& medical training

CREATIVE DIRECTOR, UI / UX, ANIMATION, VISUAL DESIGN, PHOTO/VIDEOGRAPHY

CREATIVE DIRECTOR, UI / UX, ANIMATION,
VISUAL DESIGN, PHOTO/VIDEOGRAPHY

CREATIVE DIRECTOR, UI / UX, ANIMATION, VISUAL DESIGN, PHOTO/VIDEOGRAPHY

Mixed Reality is an emerging tech that overlays virtual objects over your environment and on top of anchored elements that you can interact with using your own hands. This is especially useful for training as it uses the context of user' actual (work) environment, it can simulate situations that are otherwise impossible to experience in regular trainings and there's no need to use controllers, unlike VR.

As creative director, I guided the team of internal & external developers, designers, marketers & interns. I'd also function as a bridge between them and the founders to translate objectives, business goals, client wishes and timelines into actionable, realistic and exexutable plans whilst building the business' brand.

Context

Behind the scenes // process

Behind the scenes // process

Context

The problem
The health and safety training sector is often seen as stagnant and unexciting, with slow innovation. While many acknowledge the importance of training for life-threatening situations, it often lacks priority. Common complaints include inflexible scheduling, high costs, and limited effectiveness. Yearly mandatory training is uninspiring, repetitive, and doesn't simulate real danger, typically reaching only a select few in organizations.

The goals

Use Mixed Reality to create realistic & immersive trainings that teach users essential life-saving skills in ways that can't be replicated with traditional trainings.
Lower the barrier to entry by making training accessible for anyone that wants to learn at any time in an organisation.
Boost learning retention, confidence, and cost-effectiveness through flexible and scalable usage.
Make sure products are user-friendly & easy to use despite the target audience's hesitance towards new tech and having interaction patterns that (almost) nobody is used to.
Create a company brand that stands out as fun, inspiring and solution-focused compared to competitors that mostly focus on the newness of the tech or feel very administrative and dry.

Comply with established guidelines and be aware of anything
that can reflect negatively on the organisation' reputation.
Ideally use the established color palette, art direction & tone of voice.
The game should be playable in both English and German.
The game should be playable within ~1h by 3 to 6 people in a relaxed
setting while still providing a clear take-away of info & understanding.
The physical element should be easy to (re)produce, resist
basic scratch damage and lends itself to be dismantled for storage.
The game should be playable in different settings & environments,
with various lighting conditions that can affect object tracking.

Use Mixed Reality to create realistic & immersive trainings that teach users essential life-saving skills in ways that can't be replicated with traditional trainings.
Lower the barrier to entry by making training accessible for anyone that wants to learn at any time in an organisation.
Boost learning retention, confidence, and cost-effectiveness through flexible and scalable usage.
Make sure products are user-friendly & easy to use despite the target audience's hesitance towards new tech and having interaction patterns that (almost) nobody is used to.
Create a company brand that stands out as fun, inspiring and solution-focused compared to competitors that mostly focus on the newness of the tech or feel very administrative and dry.

The process

Agile development, two-weekly release cycle. Given how unfamiliar user were with the interactions, we quickly realised it was best to keep actions & instructions as simple, short and bite-sized as possible without losing crucial information.
Continuous user testing with user groups, both target audience and clients. Included journey mapping, (paper) prototyping, focus groups, usability testing and A/B tests.

Comply with established guidelines and be aware of anything
that can reflect negatively on the organisation' reputation.
Ideally use the established color palette, art direction & tone of voice.
The game should be playable in both English and German.
The game should be playable within ~1h by 3 to 6 people in a relaxed
setting while still providing a clear take-away of info & understanding.
The physical element should be easy to (re)produce, resist
basic scratch damage and lends itself to be dismantled for storage.
The game should be playable in different settings & environments,
with various lighting conditions that can affect object tracking.

The outcomes

Created multiple MR applications for both whitelabel and client-specific use-cases, incl. schools, businesses & hospitals. They reported a faster pace of learning & better staff performance.
After extensive user-testing, they expressed they experience increased learning retention, confidence in their actions and feeling empowered to act compared to regular trainings.
Managed to reach the national news multiple times.
Became official Microsoft Silver Mixed Reality partner.

Comply with established guidelines and be aware of anything
that can reflect negatively on the organisation' reputation.
Ideally use the established color palette, art direction & tone of voice.
The game should be playable in both English and German.
The game should be playable within ~1h by 3 to 6 people in a relaxed
setting while still providing a clear take-away of info & understanding.
The physical element should be easy to (re)produce, resist
basic scratch damage and lends itself to be dismantled for storage.
The game should be playable in different settings & environments,
with various lighting conditions that can affect object tracking.

AR, VR, and MR?

Hot damn, there's a lot few terms nowadays. No worries, check out the video to see what the difference are!

Hot damn, there's quite a few terms nowadays.
No worries, check out the video to see what the difference are!

Hot damn, there's quite a few terms nowadays. No worries, check out the
video to see what the difference are!

In a nutshell

Augmented Reality
AR puts a digital layer over reality that you control through an interface, like a phone or tablet.

Augmented Reality
AR puts a digital layer over reality that you control
through an interface, like a phone or tablet.

Augmented Reality
AR puts a digital layer over reality that you control through an interface, like a phone or tablet.

Virtual Reality
VR shows an entire virtual world, taking you out of reality. You will need to use controllers to interact.

Virtual Reality
VR shows an entire virtual world, taking you out of reality.
You will need to use controllers to interact.

Virtual Reality
VR shows an entire virtual world, taking you out of reality. You will need to use controllers to interact.

Mixed Reality
MR blends digital elements with the real world, which you can interact with using your hands.

Mixed Reality
MR blends digital elements with the real world,
which you can interact with using your hands.

CPR+

Would you know what to do if someone had a cardiac arrest right now? 🤔 If not, you’re not the only one. Enter CPR+, the Mixed Reality course that teaches you all the steps to perform a resuscitation, whenever and wherever you want.

The HoloLens 2 projects a lifelike digital victim over a CPR dummy. The app guides you through each step, while keeping track of your progress to share feedback along the way.

Fire+

If a fire would start near you, would you know how to act? 👩‍🚒 A fire occurs more often than you might think. With Fire+ you'll learn how to deal with different fire types and fire extinguishers in a playful way.

Through guidance or by experimenting yourself, you learn what works, what doesn't, and why. You see the consequences of wrong choices to learn what the impact can be. For example, throwing water on a grease fire is quite… exhilarating.

Vital+

Training medicall staff on realistic scenario's is limited, given a lot of accidents & conditions can't be fully replicated by actors. With Vital+, the ABCDE method is trained using mixed reality, which overlays a victim on top of a dummy with sensors.

A trainer can trigger interventions & monitor how staff is performing on the patient. This way an entire procedure can be trained, incl. swellings, increases in blood pressure, fainting, etc.

Keep on innovating!

Highfive for reading the whole thing!

Wow, you made it down here!

If I can help you with a similar project, let's talk.

Get in touch