Links from Adi Stephan - Assessments in Virtual Reality, Where, When and How?

• The Miro notes from this session 
• Connect with Adi on LinkedIn
• Read through 'Seeing is believing'
• Check out the Fire Extinguishing Simulator

Automatic Transcript of Adi Stephan - Assessments in Virtual Reality, Where, When and How? 

[Adi Stephan] 18:00:02
Thank you for that. Yes, coming to you, hopefully very live from Johannesburg, South Africa where we’re based.

[Adi Stephan] 18:00:13
And I'm hoping to show you a little bit what we doing on the assessment side, and specifically assessments in virtual reality, and the virtual reality topic and we were just talking about that is is very close to my heart, like you were saying, I'm not

[Adi Stephan] 18:00:31
a technologist, I'm actually an HR personnel several HR degrees.

[Adi Stephan] 18:00:38
And I'm looking at this always from an HR point of view, and I'm going to try and see how I can bring this technology and the assessments a little bit closer and to show you what is actually available in the virtual reality space when it comes to assessments.

[Adi Stephan] 18:00:55
So, Let me jump in.

[Adi Stephan] 18:01:01
It my screen to move so quickly Lana for today understanding the narrative of the little bit of an intro. I'll talk about assessments that we on the same page, a little bit of background dictionary for today, I want to cover how what and way those questions,

[Adi Stephan] 18:01:16
then pleasure and then if there are questions. I'm hoping that we've got enough time that if there are questions that we can have a nice discussion afterwards.

[Adi Stephan] 18:01:29
Around assessments in virtual reality. So let's get started. So a little bit about myself. I'm Chief learning and Innovation Officer at the boiler room we're based here in Johannesburg, South Africa, we do have an Australian connection that I will tell

[Adi Stephan] 18:01:43
you in a moment. 25 years of experience in the learning space, and I've got probably about 1213 years of experience in virtual reality boiler room, innovation and consulting hub.

[Adi Stephan] 18:01:58
We changed the face of learning forever. And as of yesterday analytics insight has rated as one of the top 10 vr companies to watch, and you've got to see more Kaka, who is Australia, the connection on the cover of analytics inside the.

[Adi Stephan] 18:02:16
Let's look at assessments in virtual reality the walkway and the how.

[Adi Stephan] 18:02:21
I think we've got the ability and I'm sure you've seen examples of virtual reality.

[Adi Stephan] 18:02:27
Quite recently and then we've had quite a bit of activity in the virtual reality field, we are now for the very first time, able to take full control of the users experience.

[Adi Stephan] 18:02:41
And really, and actually for the first time. Immerse that person in the experience giving them that full virtual environment, are they in a Windows be all in a headset environment.

[Adi Stephan] 18:02:52
And for the very first time, we've got the ability to fully assess the learner, and I'll explain that just now.

[Adi Stephan] 18:03:01
And with that, having the ability what we've got as well, when we design these experiences. Now, we can design the assessment component of this, that we actually get the learning results that we want, that's measured assessed and correctly scored, because

[Adi Stephan] 18:03:21
for the first time we can actually assess this almost in the real world, and I'll show you what what we've got available and what we can do.

[Adi Stephan] 18:03:34
So, I'm going to be using a couple of words today, when I talk about the experience.

[Adi Stephan] 18:03:40
I normally talk about experience in two ways as an event or a game, but it's also the skills and knowledge guide by completing working a set of components in virtual reality.

[Adi Stephan] 18:03:53
Then the concept of virtual reality can refer to other Windows based experience so we've got a number of experiences that are working by normal tablets or cell phones or computers being a laptop or a desktop PC, where you go into a virtual reality experience

[Adi Stephan] 18:04:14
and you controlling the experience with your Ws geeky.

[Adi Stephan] 18:04:18
And then obviously the virtual reality headset that you put on where you've got the ability to be even more immersed and you actually feel like it's totally real like you're in it.

[Adi Stephan] 18:04:29
Then assessment, I think we all know skills or knowledge check point that has a measured value assigned.

[Adi Stephan] 18:04:35
We'll talk about that in quite a bit of detail. And then, I'm using the term controllers and I'm going to show you a couple of the controllers the VR headset or window screen allows you to see the virtual world controllers allow you to interact with it.

[Adi Stephan] 18:04:48
And I think the controllers is a common important concept here, purely because it makes the experience even more realistic controllers help you register movements in a virtual environment, and they allow us to actually interact with the experience.

[Adi Stephan] 18:05:07
Okay.

[Adi Stephan] 18:05:10
So, I think, from an assessment point of view I think this is all nothing new to you.

[Adi Stephan] 18:05:16
When we talk about assessments and this is just a very quick recap the assessment is a checkpoint to determine if the skills or knowledge required by the individual have actually been achieved by the learner, we just doing a checkpoint, and we see in

[Adi Stephan] 18:05:32
can the person do what we've set out to do the VR learner cannot demonstrate the competency competency or group of competency, to a specific and required stand.

[Adi Stephan] 18:05:45
We've got the formative assessments, think we all know determines the learners knowledge and skills including any gaps, as they progress through the immersive experience.

[Adi Stephan] 18:05:55
We can build these in and a lot easier to build in then with any of the other traditional learning experiences.

[Adi Stephan] 18:06:02
We can use that to guide the experience, I he. When we do this, we can get to the point where we can say to the learner, you quite haven't mastered this yet.

[Adi Stephan] 18:06:15
Will you please go back and redo this we can do this in a way that is actually very subtle and, in some instances the learner doesn't even realize that he's been redirected.

[Adi Stephan] 18:06:26
And then the formative assessment occurs during the immersive experience, then we do the summative typically at the end of the section or task where the end of the experience.

[Adi Stephan] 18:06:35
And then we try and determine the level of understanding the beer Luna has achieved, and this is called. Now for the first time and I'll show this to you.

[Adi Stephan] 18:06:43
When we score this, the amount of information that we've got available to score this is absolutely incredible. We've never had this type of information before.

[Adi Stephan] 18:06:54
And then obviously but we can we can evaluate individually against against the set standard and or outcomes.

[Adi Stephan] 18:07:01
We can also do a whole bunch of things like multiplayer etc.

[Adi Stephan] 18:07:08
So, and I love my animation Dave for the very first time, the assessment can be almost 100% alarm to the workplace environment. Okay.

[Adi Stephan] 18:07:20
We've never as learning professionals had this before, that the assessment can be 100% aligned to the work environment.

[Adi Stephan] 18:07:28
Okay. any of the traditional methods that we've got available, and we think like online learning, learning, multiple choice questions.

[Adi Stephan] 18:07:38
We've got very limited application ability they to relate that back to the, to the workplace.

[Adi Stephan] 18:07:45
What we actually want to do not be wanting to do is learning professionals forever. The ultimate assessment is to evaluate the learner using real world situations.

[Adi Stephan] 18:07:59
So that can actually solve real world work problems. Okay. Now if you think about that, e learning by asking a couple of multiple choice questions, filling in a blank, using a matching drag and drop.

[Adi Stephan] 18:08:10
It's not, it's not real world. Okay, it's not in any way shape or SAS perform being able to be related back to the workplace. Okay, we want that ability to relate this back to the workplace and make this absolutely realistic.

[Adi Stephan] 18:08:26
So what we've got for the very first time with virtual reality coming along, is we can actually assess an individual by doing okay. We can allow the individual to do the task at hand.

[Adi Stephan] 18:08:40
And that's where the whole VR assessment differentiate actually comes in.

[Adi Stephan] 18:08:46
And I think we'll all probably agree that we can be assessed based but all senses are engaged. When we combine sight and sound with the relevant movement as required by the task at hand.

[Adi Stephan] 18:08:59
Okay.

[Adi Stephan] 18:09:00
As humans, we are given that visual the sound, and we do the movements. And yes, now I can demonstrate that combination is absolutely essential. So what have we got available in a virtual reality environment, what we can do, we can create.

[Adi Stephan] 18:09:19
Absolutely.

[Adi Stephan] 18:09:21
Assessment environment.

[Adi Stephan] 18:09:23
Now, for the very first time, and I'll go into a little bit of detail just now.

[Adi Stephan] 18:09:33
We've actually got the ability to allow the learner to see the consequences. And we've got the ability to allow the learner to see what incorrect actions actually looked like.

[Adi Stephan] 18:09:46
And I've got an example to see when you do something incorrectly. This is what happens. And we all know when, when do we learn best when we actually see the consequences we can experience the consequences, and we can then see, it's actually ok and this

[Adi Stephan] 18:09:58
environment, to make a mistake, itself, nothing is going to happen, but I see the consequence of my action for the very first time now, assessments are true to work.

[Adi Stephan] 18:10:09
OK, I can. I've got the ability to replicate the work environment and assess the person in the actual work environment through VR, to be able to demonstrate to me that I can actually accomplish the task.

[Adi Stephan] 18:10:24
Now this is obviously totally scalable from one player to player for player to any type of environment. And I always say the sky is absolutely the limit yet.

[Adi Stephan] 18:10:39
It provides an absolutely realistic skills assessment.

[Adi Stephan] 18:10:42
It collects all the key assessment metrics we've got so much data available at the moment that we can actually pick and choose what we want to see and I'll and I'll show you some of the matrix that we've got available.

[Adi Stephan] 18:10:54
And then we can totally and absolutely seamlessly. Get the outcomes, and the results and they are absolutely 100% interfaced with relevant learning platforms.

[Adi Stephan] 18:11:06
So at the end of the day, you've got the schools that are seamlessly pass through. And you can see exactly how individuals on a specific assessment.

[Adi Stephan] 18:11:18
So, when we use virtual reality in an assessment environment actually brings the workplace closer and it makes the assessment toward you real people will feel as if they are they and they are executing the task.

[Adi Stephan] 18:11:35
Now the interesting part is, if you have a look and then there is a study about PwC if you go to Google and you just type in PwC virtual reality you will actually get the study.

[Adi Stephan] 18:11:47
They've got some really really impressive numbers they. We, the effectiveness of virtual reality in that specifically tested this on soft skills in VR, where individuals feel that 275% more confident when they go back into the workplace to apply the skill

[Adi Stephan] 18:12:07
skill that they've just learned.

[Adi Stephan] 18:12:10
Now, you can ask yourself the question why 275% more confident. And for me, when I unpack this and when I look at this and we work with us.

[Adi Stephan] 18:12:21
The answer is actually very easy. The people are more confident to actually do the task at hand, purely because they've done it before. Okay, the individual fields that have done this before, and pain, this has actually become second nature, the muscle

[Adi Stephan] 18:12:36
memory kicks in and it feels like I've done this before. So, I'm the assessments feel absolutely, totally real.

[Adi Stephan] 18:12:45
So let's have a look a little bit at the how.

[Adi Stephan] 18:12:51
So, when we use virtual reality within a corporate learning environment. We've got a couple of opportunities that we haven't had previously, where we can create this that there are assessed in an ultra realistic and authentic environment.

[Adi Stephan] 18:13:10
Now, the authenticity of the experience and the assessment is very much dependent on how the learner provides input.

[Adi Stephan] 18:13:18
And we've got five basic ways of inputting information and sort of providing feedback on the assessment to the system for and we can use controllers, we can use custom controllers.

[Adi Stephan] 18:13:32
We've got a concept that's cool tactile VR that I'll explain in a little bit more detail now we've got hand gestures. And then last but definitely not least your standard keyboard or touch screen where we can get individuals to interact with the experience.

[Adi Stephan] 18:13:49
Now, what you've got your is a set of controllers.

[Adi Stephan] 18:13:56
There's a standard but with these using this Oculus quest to. You can see him completing a PP selection day which is a checkpoint and one of our experiences.

[Adi Stephan] 18:14:07
He needs to correctly, select the PPD and you can see some agreements have already as he selects something incorrectly, with other turns green if it's correct already if it's incorrect selected, but he's using those controllers to make a selection.

[Adi Stephan] 18:14:24
Now when I said custom controllers.

[Adi Stephan] 18:14:27
Here we've got basically as well he's actually operating, which is a sandwich machine or two. Right. And these are controllers that are the actual controllers of the machine.

[Adi Stephan] 18:14:37
So, we've built an entire experience for the American way. There is the assessment of the vehicle. We are actually driving the vehicle underground, you need to demonstrate that you've got the ability to control the vehicle and you using the actual two

[Adi Stephan] 18:14:56
joysticks or the two joysticks that you see they are custom built the top part is the actual handle of the service to a machine and Wisley then users the controllers as if he was driving the real machine.

[Adi Stephan] 18:15:12
And again, he is seeing the experience in the headset. We're just watching him on the monitor to actually see what he's saying.

[Adi Stephan] 18:15:23
I said, Take it all as one of the other ways to provide input. Now, if you've ever put out the fire, you will know that that fire extinguisher wise around 14 kilos.

[Adi Stephan] 18:15:36
Yes, we could have done this with just the controllers, but when you've just got the controllers and your hand.

[Adi Stephan] 18:15:44
It's not the same experience as being able to maneuver 14 killer fire extinguisher, which you have to use to put out the fire. Now what Andrea did they she used the controllers to control the experience again we what you're seeing on the screen here is

[Adi Stephan] 18:15:59
she seeing in the headset. And what we've got here you actually seeing a snapshot of the step by step instructions that she had to complete in order to successfully put out the fire, and you can see very clearly here in the video.

[Adi Stephan] 18:16:16
One of the tasks actually had a reg ex, where she did not complete the task correctly. Now these custom we object our controllers we've used on a number of experiences from boring to firefighting anytime of interaction where we can actually replicate

[Adi Stephan] 18:16:34
a real world component.

[Adi Stephan] 18:16:39
And we can bring that into VR we try and bring this in. And then when we do the assessment and the individual actually puts out the fire they need to go through the eight steps on steps depending on the process plant dependent.

[Adi Stephan] 18:16:53
They didn't demonstrate by using the actual fire extinguisher with the controllers attached.

[Adi Stephan] 18:17:00
This one is absolutely my favorite.

[Adi Stephan] 18:17:03
So advances quite recently with a quest to have allowed us to use hand gestures.

[Adi Stephan] 18:17:13
Again, this is for one of the mining groups where Wisley is using hand gestures to navigate the experience. Now, we've got absolute full control in creating the hand gestures, and it creates a whole amount of opportunities, not only around the navigating

[Adi Stephan] 18:17:33
for the assessments again the individual has got the ability to show us that they exactly know what to do. Now, let me use an example.

[Adi Stephan] 18:17:44
There are five goals and you have to close off one.

[Adi Stephan] 18:17:48
Okay, I can do that in a multiple choice question you have to take on which one you're going to shut off, or with VR hand gestures, you actually need to demonstrate by walking up to the valve and closing the bell for me, possibly would be an anti clockwise

[Adi Stephan] 18:18:04
one so we can exactly see whether or not, with a hand gestures you're using the correct hand gestures.

[Adi Stephan] 18:18:13
When we do the assessment, interesting one and I'll just throw it out there we currently busy with an experience for crime triggers that use quite a large amount of hand signals to signal the crane operator what to do.

[Adi Stephan] 18:18:30
Now this lends itself, absolutely brilliantly. And all of a sudden when we do the assessments, the rigor has to demonstrate exactly how he's going to just show to the crime to execute a certain task.

[Adi Stephan] 18:18:44
It does not become any more real than this when we do assessments.

[Adi Stephan] 18:18:49
Last but not least, we spoke about WASD. Here's Andrea and you're looking over our shoulder, she's using the WSD key on the keyboard to navigate around the experience.

[Adi Stephan] 18:19:05
Typically we assign the EQ to pick up objects etc and interact with the environment. Now, this one again is quite an easy one Wi Fi and we've got input, and we can do assessments, you can execute the task in the Windows environment.

[Adi Stephan] 18:19:23
Just wanna go briefly and have a look at when we design these things.

[Adi Stephan] 18:19:33
Specifically, now what we need to be aware of and what we need to look out for when we create assessments, what design elements are actually important.

[Adi Stephan] 18:19:47
Now, The first one is one that's absolutely critical.

[Adi Stephan] 18:19:52
The environment with assessment must be realistic to be immersive that believability is absolutely key.

[Adi Stephan] 18:19:59
You cannot just create a two three component environment, and then expect the assessment to be believable, the learners not going to fall for that.

[Adi Stephan] 18:20:13
Then when developing the assessment, you do need to think 3d spatial awareness 316 any direction. Let the learner engage walk around and then execute that task don't restrict them and make this to guided to enforced to driven, that the learners and you

[Adi Stephan] 18:20:32
one way to execute this because then you're not actually assessing him on his ability to actually carry out the task, which ties into the third point where, in virtual reality the to be assessed toss can be replicated in the most my new detail.

[Adi Stephan] 18:20:49
The amount of tools, the way you execute the way you stand, the way you engage, everything can be replicated in absolute immersed minute a teacher.

[Adi Stephan] 18:21:02
Thanks for so the assessment to be true to the real world ensure that all the interactivity is included. So, when we build these we need to make sure that all the interactivity is included and we do not take any shortcuts.

[Adi Stephan] 18:21:16
Then, I've seen this many many times yes even though assessment needs to be according to scout. And what I mean by that is that when you engage and you are being assessed that now all of a sudden we cannot make the valve that you have to turn off much

[Adi Stephan] 18:21:31
bigger, it needs to be realistic to scale if this is how big it is. That's how big it is that needs to be in the assessment. So everything needs to be according to scale even the assessment so we're not making things bigger we're not trying to simplify

[Adi Stephan] 18:21:44
the process but working with a scale, it needs to be according to scale.

[Adi Stephan] 18:21:52
Then one that I'm very passionate about, is use sound probably appropriately during the assessment components.

[Adi Stephan] 18:21:59
We spoke earlier on, we are said, we love it when all our senses are engaged. Now, there is no environment which is absolutely completely solid there's always a environment of guys got a specific sound what it sounds like.

[Adi Stephan] 18:22:16
Now that sounds, and all the sun's need to be brought in when we build assessments, absolutely key that you've got the right sounds that when you flip something open and it folds over it needs to make the songs.

[Adi Stephan] 18:22:32
Now, absolutely key to make that assessment absolutely believable.

[Adi Stephan] 18:22:37
Now, absolutely key to make that assessment absolutely believable. Then, another one ensure users can freely move around during the assessment, they do not need to be guided. Okay, we want to assist them in a way that they can demonstrate that they can complete the task at

[Adi Stephan] 18:22:50
the task at hand. And if we not allow them that freedom to move around. And then we actually guarding them, then the result that you're going to get out.

[Adi Stephan] 18:23:07
This past potentially not actually a true reflection of the ability of the individual. One that I absolutely love, and that we've got available assessment components can be totally randomized so I can throw these in to happen anywhere in the experience

[Adi Stephan] 18:23:19
at any point in the experience, and we can actually see how the person reacts to being assessed on this specific component. So, let's take for example one that when we try to operate a how to drive the vehicle.

[Adi Stephan] 18:23:35
One of the things that typically gets discussed and demonstrate demonstrated in VR only is what do you do when there is an engine file or a cab file for that matter.

[Adi Stephan] 18:23:49
Now, these assessment components we can spring on the individual in a in a way that they do not expected, and then the reaction is based how they would typically probably react in the real world.

[Adi Stephan] 18:24:04
So Dudley randomize the individual is still driving, he stops and all of a sudden there's smoke. What do you do okay they need to follow the correct procedure.

[Adi Stephan] 18:24:13
Now, how to handle that specific situation.

[Adi Stephan] 18:24:16
And then design the experience with a required data sets in mind for the assessment, so you need to decide that upfront, which datasets Do you want to report on, and those are the ones that you need to bring into the assessment.

[Adi Stephan] 18:24:30
So how long did it take me after I realized that the engine is on fire before I took the first step, did it correctly execute the process if there's an eight step change that process was are correct and executing the process so what data do I want to

[Adi Stephan] 18:24:50
on is absolutely key and there is a slide on this a little bit later. So those are the design elements that sort of formed the basis of creating an assessment in a virtual reality environment.

[Adi Stephan] 18:24:59
So let's understand very quickly what has actually changed.

[Adi Stephan] 18:25:04
And what has changed that this is no longer about just the knowledge is assessment.

[Adi Stephan] 18:25:10
So we can assist the learner on the actual task at hand. We can simulate all that in VR environment is completely authentic.

[Adi Stephan] 18:25:21
Then the assessing of the learner can happen on a continual basis throughout the experience is, and it's not necessarily centered or clustered AT THE END block we know from some of the other delivery methods.

[Adi Stephan] 18:25:33
So we can build this in that the learner demonstrates his ability throughout the experience, and on a continuous basis.

[Adi Stephan] 18:25:41
And then

[Adi Stephan] 18:25:44
we can increase complexity as in when we deem it necessary.

[Adi Stephan] 18:25:53
So, we can make this more difficult lyst difficult subject to the learner being able to complete the assessments correctly.

[Adi Stephan] 18:26:02
Then interestingly enough mentioned that before it's in the yellow block at the bottom they technology allows us to have multiple players in the assignment assessment environments from locations remotely across the world.

[Adi Stephan] 18:26:13
So not only can we have multiple participants to the assessment. So think of a team having to work together and you being able to assess the team. Not only can you assist the team we can have an assessor in the room we can have an observer in the room

[Adi Stephan] 18:26:28
we can have the facilitator in the room, etc.

[Adi Stephan] 18:26:32
Some of the things that we we have seen changes from traditional assessment methods via allows us to create that safe assessment environment, the learner, we can actually now for the first time, assess on situations and scenarios that are totally relevant.

[Adi Stephan] 18:26:54
They're realistic. And then this is the important part, sometimes even too risky, or too dangerous to undertake in the real world.

[Adi Stephan] 18:27:03
I mentioned the engine far but I think if you let your imagination go, there's a whole bunch of other scenarios and situations and assessments that you can think of that you can put the learner in where potentially It's too dangerous and risky, and with

[Adi Stephan] 18:27:21
virtual reality is absolutely not a problem.

[Adi Stephan] 18:27:25
Then assessment data that can out that ultimately can be measured assist and scored, and I'll show you some other detail just now. Then you can conduct the assessment a lot more cost effective.

[Adi Stephan] 18:27:39
Now, for the data that we get your return on investment. It is a lot more cost effective. I think one of them is soft a virtual reality, always needs to cost more, not entirely true.

[Adi Stephan] 18:27:52
And we can conduct assessments a lot more cost effective. I think with a much higher accuracy that we can get up.

[Adi Stephan] 18:28:03
Then I said that and I'll say it again obtaining the assessment data like never before. And then the last one for the very first time we can allow the learner to engage in decision making.

[Adi Stephan] 18:28:16
Learners must apply the learning into the decision making assessment and experience the impact of their choices are you going live, are you going rock, and you will immediately be able to see if you go live what happens and if you go rock what happens.

[Adi Stephan] 18:28:30
And sometimes we do show positive and negative outcomes if the learner max the wrong call.

[Adi Stephan] 18:28:40
Okay we spoke about the

[Adi Stephan] 18:28:46
using the platform and aligning the training team to report on various sets of data. And I said, we've got data available. black we've never had before.

[Adi Stephan] 18:28:57
And this data is by individual by group by experience, and you've got the ability to customize that as well. Now the data that we get, and that will come out of anyone or the experience is decision and immersive data.

[Adi Stephan] 18:29:12
So decision digest throughout the experience when we assess this individual, they will give us decision data, those are decisions and choices made by the learner, and the decision by the learner, on the right and the role of the situation that is where

[Adi Stephan] 18:29:28
you need to provide an outcome, and the learners got several ways to provide input. Now, one of the things that's interesting. In some instances, there is not always a right and a wrong answer.

[Adi Stephan] 18:29:41
There might be four right answers, and one wrong answer. And then again in virtual reality. We've got the ability when we assess the individual now, that they can demonstrate to us any one of the four and then see if he does go down the garden path with

[Adi Stephan] 18:29:58
the wrong answer.

[Adi Stephan] 18:30:00
You will then immediately see the consequences.

[Adi Stephan] 18:30:03
How do we do that.

[Adi Stephan] 18:30:05
We've got the relevant hand movements tracked by the controllers or pedals. So the input methods that you've seen there, we are literally able to record, manage understand every single pedal movement and control the movement the individual max.

[Adi Stephan] 18:30:25
So what that means, if you're doing somebody in a car did they break correctly that they break soon enough that they break too late, so we can when we do the assessment, and we've got those parameters.

[Adi Stephan] 18:30:48
feedback, your assessment was not successful because of the following points. I showed you the hand tracking and the controllers.

[Adi Stephan] 18:30:51
We can do device tracking the users position and the associated movement. We are they experience what are they doing, we can do hit movements, we could look and see where they're looking through speech recognition, etc.

[Adi Stephan] 18:31:07
So, with assessments, the learner as they do in the real world execute the activities tasks and associate assessments with the relevant decisions being made receiving instantaneous and I think that's important.

[Adi Stephan] 18:31:23
You saw Andrea there with a fire extinguisher instantaneous feedback on decisions made possible consequences and of course the scope, so you go through this, you demonstrate your ability, and you see the results are successful or not successful you have

[Adi Stephan] 18:31:39
been.

[Adi Stephan] 18:31:42
So we said we measure the learners movement we measure their behavior throughout the experience and all throughout the experience this has changed. So we've got assessment information available on hand, head, I and Linda movement.

[Adi Stephan] 18:32:02
For example, we can see how long somebody was in the specific scene, did they move correctly that they walk past that that that come back exactly what they did.

[Adi Stephan] 18:32:13
So duration in a specific scene. We can time box the scene. We can Tom box the assessment, then time taken to complete is also measured, again we can put people under pressure yet.

[Adi Stephan] 18:32:27
So, and I think that's an important one that displaying your knowledge or skill is one thing.

[Adi Stephan] 18:32:32
Can you do it in a moment when it matters. Okay, that part when the engine catches fire or when something happens and assessments in VR able to determine this.

[Adi Stephan] 18:32:45
So we can, when we create these assessments and VR,

[Adi Stephan] 18:32:50
create unnecessary pressure. And because the users and the ability to make the users feel present in the moment. And they're actually immersed in the experience.

[Adi Stephan] 18:33:05
So, we can create that pressure, and we can make them react as they would in real life.

[Adi Stephan] 18:33:14
So here's just an example of a couple of assessments. So this specific one here.

[Adi Stephan] 18:33:21
You had to walk around and you spot. This is a Windows environment. As you walk around you spot a couple of

[Adi Stephan] 18:33:33
things that are wrong in this hospital environment and you click on them, and it will then give you a score at the end. So basically walking around identifying areas where things are not as they are supposed to be.

[Adi Stephan] 18:33:49
I've shown you here is a PP one.

[Adi Stephan] 18:33:53
This is a knowledge check.

[Adi Stephan] 18:33:55
So this is your character you have to dress your character in the correct PP, you will see the rain and greens at the bottom where you've got something correct semi correct or totally incorrect.

[Adi Stephan] 18:34:08
Typically when it comes to this, pp. We, we would make the user go back and repeat the specific check.

[Adi Stephan] 18:34:17
We allow them to go on. But what we show the users what they should have selected what would be the right choice.

[Adi Stephan] 18:34:25
He has a specific instance where we've got a process you have to follow. So you have to press the button the file goes off, and then you've got a situation where you need to ascertain what type of fathers is, you are talking about at the top, where you've

[Adi Stephan] 18:34:44
got a specific time frame that you have to complete this in, and you have to select obviously the right fire extinguisher. You can select the wrong one.

[Adi Stephan] 18:34:54
And then again, at the end, you get your score, and the system will actually tell you, did you execute the task correctly, and you will get a score.

[Adi Stephan] 18:35:08
Then one that's a little bit more elaborate.

[Adi Stephan] 18:35:12
This is an underground experience. one of the machines for angler.

[Adi Stephan] 18:35:18
The specific machine is a Brock, if I remember correctly Brock Brock breaker. and what you need to do here you actually need to complete the entire checklist that checklist makes you go over the machine, in all the relevant detail, and then confirm whether

[Adi Stephan] 18:35:39
or not this machine is ready for operation.

[Adi Stephan] 18:35:43
What we do in the background. And this is where the.

[Adi Stephan] 18:35:47
This chick actually becomes a lot of fun, is we do break a lot of things in the background. So, when you check the machine. The first time everything might have been okay to put the machine into operation.

[Adi Stephan] 18:36:11
actually broke. So we break the trailing cable the triggers come out hydraulic oil level has dropped. So, we create a situation where you need to take action and understand that the machine is not ready for operation.

[Adi Stephan] 18:36:21
Something is not the way it should be. And you need to take the necessary corrective action to put the machine back into service.

[Adi Stephan] 18:36:30
So I said to you a little bit earlier on that actions have consequences and assessments in a VR immediately show the learner consequences of the actions actions and in particular incorrect actions, having some into severe consequences.

[Adi Stephan] 18:36:49
And what we do, we do show both the upside and the downside. So, this is part of an actual learning experience where the correct action was not followed.

[Adi Stephan] 18:37:05
And from a outcomes perspective we do show the upside and we show the downside. The downside in this instance is one of your colleagues was actually killed.

[Adi Stephan] 18:37:16
So, I think, assessments the way, and I think we all know data is everything.

[Adi Stephan] 18:37:25
And I've given you a bit of an indication of where we are with data and virtual reality and assessments. We can check Tom and Corona Corona launch ci check movement through space track assessment interactions and check what is being looked at.

[Adi Stephan] 18:37:43
So, every single piece of data that you can imagine, is available.

[Adi Stephan] 18:37:48
So, just maybe a little bit around the decision assessment data.

[Adi Stephan] 18:37:55
We, we giving the individual the opportunity in the assessment to demonstrate was the correct answer chosen was the action to a challenge completed correctly.

[Adi Stephan] 18:38:07
Did the learner show improvement, what consequences was the learner subjected to was the process sequence correct so I did the execute the process, and then detail of learner errors made from an immersive immersive assessment data perspective we get the

[Adi Stephan] 18:38:25
head, I envision a movement. We measure the learner behaviors are harder to engage with the experience the duration and time to completion. The amount of players in the experience that everybody engage harder to engage way that they engage and then we

[Adi Stephan] 18:38:40
can track location and device data as well.

[Adi Stephan] 18:38:46
So, I think data is everything so we've never had this much data available compared to traditional assessment methods.

[Adi Stephan] 18:38:56
The immersive environment captures far more comprehensive data as you can see that then can be analyzed and reported on for the individual learner. We typically when we create the assessment sit with a team before hand and say okay well what data do we

[Adi Stephan] 18:39:12
We can with quite a bit of certainty say whether or not the data appliance.

[Adi Stephan] 18:39:20
want out.

[Adi Stephan] 18:39:22
By analyzing them whether or not the envisaged learning results were actually achieve by the learner.

[Adi Stephan] 18:39:28
And then looking at this giving an opinion as to what the learner achieve, then this is a very very exciting one for me. Now, again I don't think we've got this available in any other of the other assessment methods.

[Adi Stephan] 18:39:46
And we've got the ability to fully replay the recorded assessment. Okay, everything the learner did demonstrated and even said as recorded equating to around 30 data points per user per second.

[Adi Stephan] 18:40:00
And what you can do the recorded session can be viewed in virtual reality or playback in a browser.

[Adi Stephan] 18:40:07
So, as a facilitator of a specific experience. I can take what the learner has demonstrated and then play back to them and say you know what well done here you immediately saw this however you missed this you did not go here, you should have engaged here

[Adi Stephan] 18:40:25
when this happened, you should have done this, so you can have a discussion with the middle afterwards because everything is recorded.

[Adi Stephan] 18:40:35
What you can do that as well as you can keep those assessments that if there is ever a question that are the past, and what school that he achieved what was he able to demonstrate, I can play that back and you can see on the side that there are some examples

[Adi Stephan] 18:40:50
do.

[Adi Stephan] 18:40:52
As I said you can use the assessment sessions in a portfolio of evidence that you can build on on the tasks that you have been assessed on.

[Adi Stephan] 18:41:02
And you've now got business has concrete replaceable proof of completion for individuals, specifically if you go to the mining sector that sometimes can be important.

[Adi Stephan] 18:41:12
So you've got proof that the individual has successfully demonstrated his ability to do the following task.

[Adi Stephan] 18:41:19
Then, the data provides an accurate representation of how Luna would perform in a real world environment.

[Adi Stephan] 18:41:26
And then also the assessment will tell how quickly and accurately the learner completed the assessment results are scored as I said immediately and feedback is provided to the learner.

[Adi Stephan] 18:41:36
And then the learner is immediately shown consequences because this, then this happened.

[Adi Stephan] 18:41:44
So, and I think we'll use some of the time now for some questions. In summary, I think when you do assessments in virtual reality it's all about the task.

[Adi Stephan] 18:41:56
Then, totally, totally forget traditional so if you've created traditional assessments in. Previously, forget how you credit and in virtual reality we create them totally differently.

[Adi Stephan] 18:42:08
So forget traditional make it absolutely real for the individual. It needs to be authentic.

[Adi Stephan] 18:42:17
Then feedback must be instantaneous and it can be instantaneous and it has to be instantaneous then combined assessment component so you can actually get the individual to execute the entire task into end important is to include consequences because this

[Adi Stephan] 18:42:33
was not done correctly this happened. Okay, the plot had to stop for or this is the consequence that you can immediately see then determine the complexity, you can make this as complex as you needed, and you can actually change the complexity as the individual

[Adi Stephan] 18:42:54
Goethe goes through it.

[Adi Stephan] 18:42:56
And then I think last but not least, the scenarios need to be realistic okay if you want the individual to be able to apply what they've learned in the work environment the scenarios need to be as realistic as possible.

[Adi Stephan] 18:43:13
And that's me.