Cheryle Walker - Authentic & Verifiable Virtual Assessments
Links from Cheryle Walker - Authentic & Verifiable Virtual Assessments
- The Miro notes from this session
- Visit Cheryle's Website
- Connect with Cheryle on LinkedIn
- Connect with Cheryle on Twitter
- Learning While Working Podcasts with Cheryle Walker
Automatic Transcript of Cheryle Walker - Authentic & Verifiable Virtual Assessments
[Cheryle Walker] 16:46:04
Let me get started. I'm going to kind of do a bit of a helicopter view not a deep dive. Because Joe cook is kind of coming after me, so she's going to do some more.
[Cheryle Walker] 16:46:17
What I'm going to be talking about is authentic and verifiable virtual assessments. So, for those of you who I haven't met, I do work a lot as Robin said in the live online space it's my passion, it's my comfort zone my home.
[Cheryle Walker] 16:46:31
But I do work in other digital technologies for learning and I'm a certified online learning facilitator with the Learning and Performance Institute in the UK, which is where I first met jack cook, who's coming after me so I'll say goodbye to Joe when
[Cheryle Walker] 16:46:42
she comes online for us after me. So good to meet you. Those of you I haven't met for about 18 months when we first went into the pandemic. I started asking everyone, how are you feeling about working and learning remotely.
[Cheryle Walker] 16:46:55
And so this is about 120 responses.
[Cheryle Walker] 16:47:00
There's not 120 responses on the screen of course this some some double ups and triple ups and so forth but a created a word cloud. And we can see that there's a variety of feelings about being working and learning remotely and I imagined when it comes
[Cheryle Walker] 16:47:15
to assessment and the way that we've had to adapt the way we create assessments, there will be a mixed feelings. Some people may prefer it some people may not so.
[Cheryle Walker] 16:47:27
So that's you know kind of this has been very interesting for me to look at this, as we've gone through Groundhog Day.
[Cheryle Walker] 16:47:34
It is indeed for a lot of faithful already. I love this some outlier over here on the right near the picture living the dream, I think that's quite, quite funny.
[Cheryle Walker] 16:47:44
So, I haven't been, I haven't been online all day I apologize for that I've actually been back for the first time in an onsite workshop running an onsite workshop which was a bit of a head spin, after all of that, and it's unusual for me to do that so
[Cheryle Walker] 16:47:57
I haven't been able to be online so apologies if I'm already going over some things that have been talked about, however I know there are some people who are just joining certain sessions and may not have been online all day, as well.
[Cheryle Walker] 16:48:09
So just to kind of delineate I'm going to be talking more about summative assessment because I know Joe coming after me is going to be talking about formative assessments so let's just refresh our minds what we're talking about their.
[Cheryle Walker] 16:48:21
The goal of formative assessment is to monitor student learning and provide feedback to both the teacher or facilitator, and the student, and that's what the brilliant Joe cook is going to be talking about how to do that in virtual classrooms.
[Cheryle Walker] 16:48:36
Coming up after me. So I thought I would focus then on summative assessment so this is where we're evaluating student learning and or evaluating student or employee performance against a benchmark or standard.
[Cheryle Walker] 16:48:53
And I know some of you have had some experience doing this during covered because you've had to had to adapt. So once I've kind of shown what I've got.
[Cheryle Walker] 16:49:01
I might call on some of you to share your experiences in adapting your assessments to virtual or live online live online.
[Cheryle Walker] 16:49:11
Yeah, virtual assessment doesn't have to be live online, we'll talk about some of the other different ways we can do virtual assessment virtual as in being not in the same place and potentially not in the same time zone as your assesses or not synchronously
[Cheryle Walker] 16:49:25
together, not live. So just a few principles of summative assessments and some of these, you might like to be more rigorous about than others, it depends on your purpose of assessment, and how you a critical credential people.
[Cheryle Walker] 16:49:42
So just thought I'd mentioned these ones which are the most kind of relevant or popular ones accessibility reliability authenticity and verifiability those tend to be the big four serious ones when it comes to virtual assessments or any assessments indeed
[Cheryle Walker] 16:50:00
summative assessments. So when it comes to accessibility.
[Cheryle Walker] 16:50:03
This is trickier in digital for sure there are still considerations even when you have people in a physical setting, but it's about creating fair and equal assessment opportunities, and situations scenarios regardless of individual digital digital confidence
[Cheryle Walker] 16:50:22
or access to a digital environment. So we do have to be aware that with some skills and some assessments, digital competency, or digital literacy is important and vital and in other cases it's got nothing to do with what we're being assessed.
[Cheryle Walker] 16:50:38
However, it may cause interference in the person's ability to demonstrate competency in what we are assessing in the skills that we're assessing so accessibility is very, very important consideration when we're in digital environment or virtual environment.
[Cheryle Walker] 16:50:56
We need to also be aware of adjusting or making allowance for situational, and environmental limitations, whether it be as or whether it be the assesses situational environmental limitations.
[Cheryle Walker] 16:51:11
And we need to be prepared to provide assistance and support where required so we may need to do some consultation and research prior to running our assessments virtually to understand the level of assistance and support that we may need to provide.
[Cheryle Walker] 16:51:25
So that were assesses are actually in a natural environment, and have the best chance of demonstrating their skills and knowledge in the assessments.
[Cheryle Walker] 16:51:37
We also need to be aware of and make allowance for cultural and linguistic diversities, some of which are a lot of which I've worked with these neuro diversity in some cases is emerging as critical as well in some environments and that's where I'm not
[Cheryle Walker] 16:51:51
as experienced but learning fast and some of you may have worked more with neuro diversity, but we're certainly being called on to consider those as well.
[Cheryle Walker] 16:52:03
When it comes to working environments, learning environments and assessment environments and performance environments.
[Cheryle Walker] 16:52:13
And then we might need to consider the availability of non digital avenues and access for assessment as alternatives when we're running virtual assessments, we may in fact be in a position that we need to provide non digital avenues and access.
[Cheryle Walker] 16:52:28
So, a few considerations there and, as some of you experienced people will know our accessibility is a is a hairy beast.
[Cheryle Walker] 16:52:44
Quite hairy at times. So let's just go over some other basics like reliability so reliability means that your assessment of the degree to which your assessment measures accurate precise and consistent have the skill or knowledge that you're assessing.
[Cheryle Walker] 16:52:53
So in some ways that's how close can you get your assessment environment, to the natural environment where someone would need to perform a task or a skill or demonstrate knowledge and virtual by means of its name virtual means not real, not in reality.
[Cheryle Walker] 16:53:09
So there's the even bigger challenge, how can we kind of hook up or combine a near real experience or environment in virtual and try to up that level of reliability of our assessments.
[Cheryle Walker] 16:53:26
Authenticity. So, being genuine and not corrupted or compromised is the definition I'm looking at here.
[Cheryle Walker] 16:53:33
So authenticity, how do we create our assessments that they are not corrupted not compromise there a genuine environment and a genuine opportunity for the assesses to perform and demonstrate their competency and skill.
[Cheryle Walker] 16:53:47
And then verifiability real relates to the assessment being or the data being trustworthy proven confirmed indisputable.
[Cheryle Walker] 16:53:57
So these are all big challenges out there and as I said depending on your environment and your purpose for assessment some of those may be absolutely critical.
[Cheryle Walker] 16:54:06
Others of them you may be a little bit more lacks or lenient with or tolerate some elasticity or some doubt around some of those but some of them are quite critical in certain environments.
[Cheryle Walker] 16:54:18
So let's have a look at the mechanisms for virtual assessment some of the different things that we have two mechanisms that we might need to put in place or work with.
[Cheryle Walker] 16:54:29
So the first one that comes to mind is password security and access. So giving your virtual assesses access to the environment you're going to conduct the assessment within keeping that environment safe and secure cyber secure, and then management of
[Cheryle Walker] 16:54:45
passwords so that you can know that only the right people are accessing that environment, and.
[Cheryle Walker] 16:54:51
And that people, not able to bring in imposter impulses, in a certain way oh have someone else login as themselves that's that verifiability. So that's kind of a real fundamental one in a virtual environment password security and access.
[Cheryle Walker] 16:55:06
And then there's identity confirmation which kind of comes off the back of that. So whether you will be providing passwords in Justin time fashion in order that they can't be shared to widely.
[Cheryle Walker] 16:55:16
What sort of mechanisms, you will use for identity confirmation, whether that be, you know, things like driver's license whether your pre confirm identity before the assessment starts or that kind of becomes the first stage or part of the assessments
[Cheryle Walker] 16:55:31
are a number of different ways you can do that.
[Cheryle Walker] 16:55:35
A good question Kim what suggestions do you have for preventing sharing an excess of passwords. The best one I've come across is that last minute password so you only issue the password for the system at the very last minute so there's not a lot of time
[Cheryle Walker] 16:55:49
or opportunity to share that password with anyone else.
[Cheryle Walker] 16:55:53
Thank you, David, somebody who's got some more technical skill than a multifactor authentication brilliant. Yes. So my understanding and correct me if I'm wrong David is is multi ways of authenticating that the correct person is logging in, so that may
[Cheryle Walker] 16:56:07
be yourself, a mile, their mobile phone number or even security questions to answer. Yeah.
[Cheryle Walker] 16:56:15
I think I've got that right. Thank you. So, yeah, very critical. We want to know that the person we're assessing is the person that we're assessing.
[Cheryle Walker] 16:56:23
Good one, I'd like someone else to answer this question actually in the chat panel JD payoff factor in data UK thing. Yeah. Really good question, particularly given up doses little gift to us in the last couple of weeks so you know perhaps that can be
[Cheryle Walker] 16:56:37
one for discussion. I'm not an expert in that either. And I would tend to rely on experts if I were running an assessment on sort of on the learning expert side.
[Cheryle Walker] 16:56:48
So another consideration here another
[Cheryle Walker] 16:56:58
mechanism is video observation. And I don't mean video observation to prove competency. I mean video observation of someone undertaking an assessment so perhaps the assessment is an online quiz, and we have the person on webcam so that we can kind of
[Cheryle Walker] 16:57:08
watch their behavior and body language. We can never be completely sure that they're not consulting other sources of information like googling the answers to things however if we have timed assessments, we can you know potentially reduce the risk of somebody
[Cheryle Walker] 16:57:28
finding the answers elsewhere so live video observation or even recorded video observation could be one way to try and monitor what the SSE is doing while they're completing their assessment and are they doing the things they should.
[Cheryle Walker] 16:57:45
Yeah, David given the doctors. The questions are being raised to whether the data should be kept beyond the time of use absolutely so how will that data be destroyed.
[Cheryle Walker] 16:57:55
and and how can we be sure that happens. Another mechanism then is time to performance or completion, this might be relevant to some types of assessment where you want people to be able to perform the tasks or the skills or demonstrate competency within
[Cheryle Walker] 16:58:09
a specific time period. So that's time bound, or you might want them to complete a quiz or complete an assignment or complete a scenario within a specific period of time so your software can manage that for you as well.
[Cheryle Walker] 16:58:26
If that's relevant. And then there's synchronous or asynchronous.
[Cheryle Walker] 16:58:31
Yeah, good question. Dave is rising again times performance can conflict with objectives of accessibility so you may need to make exceptions there, or consult based on, you know what, what, particularly neuro diversity is I think also so yeah, there may
[Cheryle Walker] 16:58:45
be a number of versions based on what you're dealing with.
[Cheryle Walker] 16:59:05
and anybody who's been through video interviewing kind of software or get a feel for what asynchronous is it can still be timed it can still be only one attempt.
[Cheryle Walker] 16:59:12
Your first attempt, those kinds of things can still be put in place but it may not be that you are live online with anybody else, it's recorded and examined later.
[Cheryle Walker] 16:59:23
And then another mechanism that you might need to think about is whether you're going to allow the assesses multiple attempts, or you're working on a first and only attempt of the assessment.
[Cheryle Walker] 16:59:35
So time bound, a number of attempts can really tighten up how you're assessing competency.
[Cheryle Walker] 16:59:42
And then when it comes to assessment software, I'm not going to recommend any particular software I do recommend gt calm, as a place that you can go and investigate software for different purposes and so a couple of days ago when I looked at the grid
[Cheryle Walker] 16:59:56
report for assessment software their fall 2022 report. This is what they are listing as the leaders in assessments software now this big the question for me this a lot of this looks like quiz based software, which is not necessarily all the types of software
[Cheryle Walker] 17:00:09
that you can use for assessment, it looks like it's focused on quiz based.
[Cheryle Walker] 17:00:14
But yeah, there's many different software that can you can use depending on how your assessments running and indeed what sort of skill or task you're assessing what you're getting assesses to do.
[Cheryle Walker] 17:00:24
And so that's why, it'll be interesting to hear Joe talk about the virtual classroom as a software for assessment.
[Cheryle Walker] 17:00:31
Okay, so let's look at some of the methods, then Hey Joe Good to see you. So nice to see you.
[Cheryle Walker] 17:00:39
So yeah, all my curly questions I'm referring to you.
[Cheryle Walker] 17:00:42
All my curly questions I'm deferring to you. Fantastic already methods in virtual assessment, and as I said at the start of the session Joe I'm letting you cover the virtual classroom I'm kind of going bigger picture helicopter view of all sorts of virtual assessment, not just the formative, yeah.
[Cheryle Walker] 17:00:56
Alrighty, what sort of methods in virtual assessment. So, the one that comes to mind immediately is quiz based on knowledge closed or open questions being able to repeat, or remember, or reference, knowledge, and bring it to the fore in the assessment
[Cheryle Walker] 17:01:13
and it might be time bound or Nashi might have a lot of time you may have restricted access to assistance and reference or you may have open book, kind of scenario where you can reference knowledge from elsewhere so quiz is kind of the, one of the common
[Cheryle Walker] 17:01:27
ones video evidence, so this may be asynchronously you know where you got a film yourself doing something or somebody else films or captures video evidence of you, performing the skill or the task or, or bringing forth your knowledge, so this is a little
[Cheryle Walker] 17:01:47
bit similar to video interviewing, that kind of mechanism into personal interview maybe another way so this is kind of the examiner sits down and asks you questions provides challenges, and then make judgments, as we're going through.
[Cheryle Walker] 17:02:06
One of the most gesture is being considered not sure that question because I might park that for now we'll come back to that.
[Cheryle Walker] 17:02:14
Yeah, so interpersonal interview is another way and often use for recruitment purposes.
[Cheryle Walker] 17:02:19
Teach back is another one that all principle that if you know the skill the task the knowledge well enough you can teach it back to somebody else and demonstrate your competency that way, that's another option.
[Cheryle Walker] 17:02:33
Then there's live stream performance or skills demonstration so that could be virtual classroom type of software or any other type of video, live stream software.
[Cheryle Walker] 17:02:45
And that makes me think about things like are you going to allow a single attempt is that time bound you have a certain amount of time on the live stream and that cuts you off.
[Cheryle Walker] 17:02:54
And there's all sorts of parameters around that that can use, and definitely any of these can be used as we said for somebody all formative assessment on kind of focusing on summative assessment and a lot of what I'm talking about but jello talk more
[Cheryle Walker] 17:03:06
about about formative assessment.
[Cheryle Walker] 17:03:10
And then observational group work, which is another one that I've done a fair bit within assessment centers as pre employment or employment selection recruitment.
[Cheryle Walker] 17:03:19
So putting people in groups with tasks in a live online synchronous environment and observing them and making inferences judgments, that kind of thing by observing their behaviors their skill, the way they interact with others the way they apply themselves
[Cheryle Walker] 17:03:36
to a task, that kind of thing is done a lot in the graduate recruitment environment, I know.
[Cheryle Walker] 17:03:43
So, that can be an interesting way as well and that's where our trusty breakout rooms become really helpful and sometimes you might need additional support from other observers so that you get a more broad range of judgments, or a more diverse range of
[Cheryle Walker] 17:04:00
web servers.
[Cheryle Walker] 17:04:01
In that case.
[Cheryle Walker] 17:04:03
And the final thing I just wanted to cover and mentioned before we, we go to some questions or some discussion we can have is an empathic practices in virtual assessment really important because if we're not, if we are in the virtual realm if we are not
[Cheryle Walker] 17:04:20
physically together with our assesses we do need to be a little bit more sensitive about building in some empathy. So one of the things I like to talk about when it comes to either learning or assessing in this environment is the concept of setting the
[Cheryle Walker] 17:04:33
table, you know setting an environment where you are welcoming people in and pardon me, I've just got some in a noisy environment at the moment but we will be fine with that in a moment.
[Cheryle Walker] 17:04:42
So setting the table you know setting up the environment so that it is not confusing so that you're not creating cognitive blocks by, you know, having things, not ready.
[Cheryle Walker] 17:04:54
You know, being warm and welcoming all that kind of thing so setting the table this is an analogy I quite like from Maryland boss savant who's the woman with the highest recorded IQ in the world.
[Cheryle Walker] 17:05:05
She said you should be able to set the table so that people feel like they're sitting and dining, not so sorry, you should be able to set the table so that people feel like they're dining, and not just sitting and eating.
[Cheryle Walker] 17:05:18
And I just think that's a wonderful analogy.
[Cheryle Walker] 17:05:21
There's other empathy objects from my own work that I've been able to demonstrate as well as defined so we use of webcams acknowledgments you know acknowledgement of country is something that we do in Australia but acknowledging other things might be
[Cheryle Walker] 17:05:37
worthwhile creating space, holding space and creating safe space for assessment, whether that be psychological space or virtual space or time, space, eye contact introductions inclusive agendas, there's a lot of things here that we can consider when we
[Cheryle Walker] 17:05:53
create the right environment for people to be able to perform at their best in assessment.
[Cheryle Walker] 17:05:59
And then finally I just wanted to have a little peekaboo at my no foul breakout rooms method that I like to, you know, breakout rooms can be threatening for people who aren't used to them so I tend to take this six step approach the first step really
[Cheryle Walker] 17:06:12
is to ignore the technology, don't worry about that for a moment. Let me explain the task.
[Cheryle Walker] 17:06:18
Or, you know what you're going to be assessed on and how the assessments going to be run as though you know the virtual part of it doesn't matter, and I'll confirm understanding with you if that, then I will allocate the breakout rooms and let you know
[Cheryle Walker] 17:06:29
who will be in the breakout rooms and what the experience will be like and how long they'll go for them to open the breakout rooms and make sure people are unmuted with webcams on.
[Cheryle Walker] 17:06:40
And then as a facilitator or assessor I would join each breakout room and just check in that everybody's Okay, things are flowing fine. We can send broadcast messages so that might be time notations or boundaries and then closing the breakout rooms and
[Cheryle Walker] 17:06:55
all the debriefing.
[Cheryle Walker] 17:06:57
Yeah. Okay, thanks Joe for that comment. So that's there. If you're interested in my book leverage live online. That's a little reference of where you can buy my book.
[Cheryle Walker] 17:07:10
And finally, if you want to connect with me on LinkedIn and continue the conversation or email me at any time, I'm open to that as well that would be wonderful.
[Cheryle Walker] 17:07:19
Alright so we're kind of at question time or if others want to share their experiences with virtual assessment that would be really helpful as well. And we can create a discussion around that so I'll stop sharing my screen just right now.
[Cheryle Walker] 17:07:33
I think Robin mentioned to me a little earlier, I might just stop in if Mick why there's there Robin said Mick might have some interesting things to share around virtual assessment that you've been working on is that right make if you're there.
[Cheryle Walker] 17:07:57
The efficient board has a lot of valuable takeaways Thank you Liam so we might go and have a look at what's on the mirror session board fantastic.
[Cheryle Walker] 17:08:03
But anybody else, not only make it does anybody else want to ask questions, or share some thoughts, or experiences around virtual assessment.
[Robin Petterd - Host] 17:08:14
Question, have a question for you, Cheryl used to sort of quizzes formats in live sessions, have you seen that work.
[Robin Petterd - Host] 17:08:26
And how does that work.
[Cheryle Walker] 17:08:28
Yeah and Joe might have some more to add in her presentation coming up.
[Cheryle Walker] 17:08:32
Yes, I think quizzes can be engaging. They can trigger trigger memory but trigger, people solving problems working on things when it comes to mechanisms obviously there's polling and web conferencing but there's a lot of better alternatives these days
[Cheryle Walker] 17:08:46
like minty meter cahoot.
[Cheryle Walker] 17:08:50
So many of those different kinds of polling or survey software, which can be anonymous then as well. So it can give you a good feel for the competency of the group without, you know, shadowing anybody in particular.
[Cheryle Walker] 17:09:04
Yeah, I think, I think quizzes good for knowledge, where it's performance of a task, definitely you would want to more swing towards observation, or video evidence or something that you can actually see the skill being performed.
[Robin Petterd - Host] 17:09:18
Yeah, it was interesting that idea of live video evidence as well as effectively we do have cameras yeah at that does open up different types of possibilities.
[Robin Petterd - Host] 17:09:29
Cool Jessica question for you, and I think Kim has got a quick comment as well.
[Cheryle Walker] 17:09:37
Yeah, I'm good point Joe I found clients are mixed responses to virtual assessment, they doubt, virtual assessment and worry about things like data, data collection, which has been discussed a fair bit in the chat panel.
[Cheryle Walker] 17:09:54
They also worry about verifiability reliability, all that kind of thing. However, we found the pandemic has forced clients to realize that that's the only way they're going to get through and they need to quickly upscale and put something together so
[Cheryle Walker] 17:10:14
yeah a lot I think a lot of people would prefer traditional on site or physical based assessments but virtual assessment has got a lot of people through in the last couple of years.
[Cheryle Walker] 17:10:22
But interested to hear other views as well to that question Joseph asked, How to clients respond to a virtual assessment versus the traditional on site or physical settings.
[Robin Petterd - Host] 17:10:37
Yeah.
[Robin Petterd - Host] 17:10:38
Exactly, exactly. And an example there that approach the probably one of the very, very realistic one.
[Cheryle Walker] 17:10:49
David says What about smaller assessments harder to get someone to do all of them. I'm wondering David whether you're talking about smaller assessments that are not compulsory, and you're trying to encourage people feel free to unmute and chat if we're
[Cheryle Walker] 17:11:02
not quite on your wavelength.
[Cheryle Walker] 17:11:11
It's interesting what Kim says to about during the pandemic. Your local university used a very resource intensive approach of using people to virtually observe exams.
[Cheryle Walker] 17:11:20
Yeah, so on camera doing your exams on camera.
[Cheryle Walker] 17:11:25
Yeah, and that feeling of snooping at people and particularly when they're in their home environment. That's quite sensitive as well you know who's watching me work in my home environment.
[Cheryle Walker] 17:11:39
Okay smaller David's out of clarity they're smaller assessments rather than doing one big exam online.
[Cheryle Walker] 17:11:47
Yeah, I'd be.
[Cheryle Walker] 17:11:49
I'd be saying yes it is harder to get someone to return to keep doing smaller assessments. However, for me it's about relevance, is it relevant that they that the one big exam tests competency or knowledge.
[Cheryle Walker] 17:12:02
And would that be realistic in the performance scenario that they would need to be able to regurgitate or recall, or perform all of those things in one big go, or would it be.
[Cheryle Walker] 17:12:14
When I think of workplace competency. more likely you've got performance support and colleagues to rely on and and that kind of thing. And so, the smaller assessments are probably realistic more as to how some workplaces flow.
[Cheryle Walker] 17:12:29
So it would depend on relevance to, to, you know what the ideal performance scenario looks like ideally assessments should create the conditions very similar to where people really have to perform, or, or provide that knowledge and skill.
[Cheryle Walker] 17:12:43
Yeah, I agree David traditional three our university exam, very outdated was in my years many years since I've been a university but you know I used to think back then what relevance is this got to anything that I'll ever do and in the workforce to have
[Cheryle Walker] 17:12:56
to regurgitate right you know three essays in three hours kind of thing it's, it's, it's proving that I've studied, but it doesn't necessarily prove how well I'll do in in a job setting, sometimes.
[Cheryle Walker] 17:13:10
Oh good, good point came on the neuro diversity yes maybe an adjustment, it may be it.
[Cheryle Walker] 17:13:16
Yeah and accessibility adjustment for neuro diversity maybe a good reason to do that.
[Cheryle Walker] 17:13:25
Cool.
[Robin Petterd - Host] 17:13:26
Any other questions.
[Robin Petterd - Host] 17:13:31
Thank you show for a really nice summary of some of the methods and techniques and some of the things unique people need to think about with online assessment.
