Links from Jinjing Zhao - Connecting Course Assessments to Program Outcomes

• The Miro notes from this session 
• Connect with Jinjing on LinkedIn
• Check out Instructure

Automatic Transcript of Jinjing Zhao - Connecting Course Assessments to Program Outcomes

[Jinjing Zhao] 11:32:02
So Hi everybody, my name is Jinjing

[Jinjing Zhao] 11:32:06
I work with my colleague Mike Sharkey. We are from Arizona State University at being united states for a disclaimer.

[Jinjing Zhao] 11:32:20
Me and Mike we're the data, guys, we're not assessments experts so we actually work together with the assessment office in our university for program assessment.

[Jinjing Zhao] 11:32:33
So, to piggyback on the breakout room questions, you know like, how do we improve the increase, improve the quality, quantity and quality of evidence in digital learning.

[Jinjing Zhao] 11:32:48
This presentation is going to be mostly about improve, like, increase the quantity of evidence for program assessment.

[Jinjing Zhao] 11:33:02
So, I'm.

[Jinjing Zhao] 11:33:02
So our goal is this, we want to gather data to answer the question, our students learning the things we said they should learn.

[Jinjing Zhao] 11:33:14
Normally we measure students learning by their GPA is of course grades, but you know those measures to course. For example, if I graduated with a. GPA of 3.8, what does that say about my skills, or if I get a programming course, what does the course quite

[Jinjing Zhao] 11:33:29
say about my skills. So we can't really use that to measure learning. But when a program goes through a program assessment or accreditation audit, it will be asked to measure specific learning outcomes.

[Jinjing Zhao] 11:33:43
But that data is not easy to gather, so in the past, you would have to gather past the students work and reevaluate them for specific learning outcomes.

[Jinjing Zhao] 11:33:54
So that's a very time consuming process and it's not scalable.

[Jinjing Zhao] 11:33:58
So to solve these problems.

[Jinjing Zhao] 11:34:01
We propose the tagging method.

[Jinjing Zhao] 11:34:04
Okay.

[Jinjing Zhao] 11:34:06
So, our premise, assess.

[Jinjing Zhao] 11:34:08
We already assess student work, when we create assignments in class.

[Jinjing Zhao] 11:34:13
If that assessment is valid. I don't want to reassess with new reviewers at accreditation time, it was, we also need a scalable solution and this is very important because there are hundreds of programs in our university.

[Jinjing Zhao] 11:34:29
So our solution is to pull out existing class assessment scores.

[Jinjing Zhao] 11:34:34
And then aggregate them for a programmatic assessment.

[Jinjing Zhao] 11:34:41
As you know, a program would have several learning outcomes for example a computer science program might have outcomes essay. While graduated from our program should be able to, to be able to code and to solve computing problems, but it could also have

[Jinjing Zhao] 11:34:58
things like graduates from our program should also be able to have the ability to if to communicate technical concepts with technical or non technical audience.

[Jinjing Zhao] 11:35:10
So that's program outcome, and on the other hand, a program also has courses that students need to take, and each course have assignments and quizzes, exams and students will be assessed, some assignments are graded are assessed beer.

[Jinjing Zhao] 11:35:44
So.

[Jinjing Zhao] 11:35:46
So in other words, for a program assessment, we want to identify where in the curriculum, the learning of different learning outcomes takes place, and where the learning is assessed.

[Jinjing Zhao] 11:35:57
So we want to associate the program outcomes with assignments or Rick Rick Ross, in our courses.

[Jinjing Zhao] 11:36:06
So making that association between program outcomes and of course assessments, is our first step.

[Jinjing Zhao] 11:36:13
Here is an example of this outcome assessment mapping. So, this is a program. This is a special sciences house education program, and it has several outcomes.

[Jinjing Zhao] 11:36:36
measure 1.1 is a students will plan assessment for health education or promotion, so that specific skill is taught in the course of http 454. And it's assessed, specifically in module one paper.

[Jinjing Zhao] 11:36:38
One of them is graduates will be able to assess need, assess needs resources and capacity for health education or promotion and under that all calm, there are three measures.

[Jinjing Zhao] 11:36:52
And the minimum score of 70 70% means they students who scored about 70% on that module one paper. Would it be considered as having mastered outcome measure 1.1, and overall the program expects 80% of all the students to have mastered outcome measure

[Jinjing Zhao] 11:37:15
1.1.

[Jinjing Zhao] 11:37:18
So this is the outcome assessment Association, plan, and a lot of programs already have a plan like this.

[Jinjing Zhao] 11:37:28
So, what we ask them to do is to add another column to their plan which has this tag or hashtag.

[Jinjing Zhao] 11:37:38
So it's like the hashtag in social media so for each program outcome. We want to give it a unique hashtag, and our method to create a unique hashtag for each outcome measure is to combine a program code was the outcome measure.

[Jinjing Zhao] 11:37:57
So at our university, each part of the program has a unique program code for this particular program. The code is an HHEH PBS is not.

[Jinjing Zhao] 11:38:08
So this would never happen in like a regular English words.

[Jinjing Zhao] 11:38:12
So by combining that code with outcome measure 1.1, we can create a unique hashtag, just for this programs outcome measure 1.1.

[Jinjing Zhao] 11:38:23
And this tax would function like Apple air tax, I don't know if you have used it or have heard about it. So Apple aired hacks can help you find things it's just a little chips.

[Jinjing Zhao] 11:38:35
Like you can buy.

[Jinjing Zhao] 11:38:36
And you can attach it to your keys to a remote, or to your dog. And you can see where they are from your phone so this hashtags outcome hashtags, the function as air.

[Jinjing Zhao] 11:38:48
Apple air tags. So it's a similar idea we put it out context in assessments in our courses, and then we this hashtag will help us find a course assessments, that's needed for program assessment.

[Jinjing Zhao] 11:39:02
So I want to sort of pause here and check was the audience.

[Jinjing Zhao] 11:39:07
If anyone has questions so far about outcome assessment mapping.

[Jinjing Zhao] 11:39:22
If not, I'm gonna move on.

[Jinjing Zhao] 11:39:29
Oh, me check.

[Jinjing Zhao] 11:39:33
Okay.

[Jinjing Zhao] 11:39:34
All right.

[Jinjing Zhao] 11:39:36
Okay. Where do we place this text message our second step for step we have a plan. All right.

[Jinjing Zhao] 11:39:43
Oh, I just want to clarify, there are three courses for the one outcome. No, so I'll come measure 1.1 is in HTTP 454 and is specifically assessed in module one paper.

[Jinjing Zhao] 11:39:56
If you look at outcome measure 1.3 as also taught in http 454. But it's assessing quiz, number one.

[Jinjing Zhao] 11:40:06
So they're in different courses and different assignments.

[Jinjing Zhao] 11:40:11
Okay. All right, Let's move on.

[Jinjing Zhao] 11:40:17
Um. So step number two, we want to put the outcome hashtag into assessments, A.

[Jinjing Zhao] 11:40:24
in camera so cameras, is the learning management management system that we use. So, I think almost all courses of the university has a Canvas course shell.

[Jinjing Zhao] 11:40:35
So, we can.

[Jinjing Zhao] 11:40:38
So, we can place a tax into an assignment or an exam. So in that case, we would have us students score for that assignment or for the entire quiz or exam, as the measure for students performance, sometimes we feel assignments or or exam score is due to

[Jinjing Zhao] 11:40:57
course not granular enough for us to capture students learning, then we can put the hashtag into just a row in a rubric. So for example, final paper might be like a big project, so it's assessing several learning outcomes.

[Jinjing Zhao] 11:41:15
So, and it's using a rubric, so we can attach different outcome outcome hashtags into different roles in the rubric.

[Jinjing Zhao] 11:41:25
I'm going to show you an example.

[Jinjing Zhao] 11:41:28
So this is an example of adding a hashtag to a to an assignment. So this course is house advocacy module one assignment is community organizing, and this is just things assignment instruction, and we're literally just add the hashtag into the instruction

[Jinjing Zhao] 11:41:47
text.

[Jinjing Zhao] 11:41:49
And here's an example of adding a hashtag to a rubric roll to the course is health promotion program planning and implementation.

[Jinjing Zhao] 11:41:58
This is their final program plan, assignment. This assignment use this rubric to grade.

[Jinjing Zhao] 11:42:05
And we want to associate outcome measure 1.8 point our I don't know what that means but the the program those attached to that outcome measure into needs assessment criteria in this rubric.

[Jinjing Zhao] 11:42:19
So we're not using students score for the entire assignment as their as a measure for their performance, we're just using their score for this particular role in a real quick.

[Jinjing Zhao] 11:42:33
Okay.

[Jinjing Zhao] 11:42:36
So, to get the data you need an infrastructures way.

[Jinjing Zhao] 11:42:42
We mostly use two types of data to do program assessments, one is learning management system data so in this case is canvas.

[Jinjing Zhao] 11:42:54
Our university actually just a Canvas data every day. So from data warehouse, I can get students scores for their assignment, their requests, and every role in the rubric.

[Jinjing Zhao] 11:43:03
And on the other hand I work with student information system data so it's like people thought data from there I can get students characteristics information like what costs are enrolling what their majors their campus, their admission type.

[Jinjing Zhao] 11:43:18
So, an ethnicity, etc so depends on what's available in PeopleSoft, and then.

[Jinjing Zhao] 11:43:28
So because we have that data in our data warehouse, I can then run sequel queries, and then create reports on Tablo.

[Jinjing Zhao] 11:43:39
So, our all of our program assessment reports that I create for the programs on tableau, and it's automated.

[Jinjing Zhao] 11:43:47
So, this is a, this is one of the reports that we create it.

[Jinjing Zhao] 11:43:53
So, this particular dashboard actually includes data for 20 or 30 programs in one dashboard so this is the view for the program of exercise and wellness.

[Jinjing Zhao] 11:44:10
And on this vo student, sorry the users can see how students perform for each up their outcome program outcomes. So for example, if you look at program outcome measure 1.4.

[Jinjing Zhao] 11:44:25
They PR expects 80% of the students, scoring, at least 75% or more on whatever assessment associated with 1.4.

[Jinjing Zhao] 11:44:37
So there are a total of 85 students enrolled in the courses that were 1.4 is tagged and 84 of them scored about 75% or more on whatever assessments associated with 1.4.

[Jinjing Zhao] 11:44:57
So that means, almost everybody has mastered program outcome. 1.4.

[Jinjing Zhao] 11:45:04
But if you look at 4.3.

[Jinjing Zhao] 11:45:06
Again the program expect everybody to against he expects at least 80% of its students to score at least 75% in assessment.

[Jinjing Zhao] 11:45:17
So there are 132 students enrolled in this course as well, but only 72 students scored about 75.

[Jinjing Zhao] 11:45:28
So that means only about half of the students mastered program outcome measure 4.3, so that gives the program, an opportunity to to look into their curriculum look at look at their course design and figure out possible ways to improve students performance.

[Jinjing Zhao] 11:45:47
And on the left side we give the users filters to work with.

[Jinjing Zhao] 11:45:53
They can choose academic year.

[Jinjing Zhao] 11:45:58
They can also choose student type so in our university. A lot of times they want to program assessment there, they're more interested in how majors are doing.

[Jinjing Zhao] 11:46:10
So, here they can choose to see everybody's performance, or just see the major students performance.

[Jinjing Zhao] 11:46:18
Okay.

[Jinjing Zhao] 11:46:21
We can also divide the results by students population.

[Jinjing Zhao] 11:46:25
So, for example, program outcome measure 1.1 dot m.

[Jinjing Zhao] 11:46:31
So that measure, you can see all my students perform much better than the emerging students are on ground students.

[Jinjing Zhao] 11:46:40
There is a much difference between female and male students, honor, college students perform slightly better than now honor, college students and transfer students perform much better than other types of students.

[Jinjing Zhao] 11:46:54
So this gives them an opportunity to look at if there's any gap in sort of in learning outcome results across different student population

[Jinjing Zhao] 11:47:10
I'm thing our dashboard. We also create this field to show the locations of different outcome measure. So, this is mainly used for quality check.

[Jinjing Zhao] 11:47:23
Basically, this tells the user for each of your outcome measure, we are able to find your tag, in which courses, English assignments.

[Jinjing Zhao] 11:47:34
So this is kind of the reality check for them.

[Jinjing Zhao] 11:47:38
They users can use this dash this wheel to compare this with the initial outcome assessment plan that they have. And if, to see if the tags are placing the cracks places or if any tag is missing from the data.

[Jinjing Zhao] 11:47:55
Because this this process involves like manually adding the packing to courses so sometimes their arrows, or sometimes a tag could get accidentally deleted by the instructor.

[Jinjing Zhao] 11:48:08
So this is mainly used for quality check quality assurance.

[Jinjing Zhao] 11:48:15
So that is how we create program outcome assessments reports. These reports are automated because the data behind the behind the dashboard is automated.

[Jinjing Zhao] 11:48:26
Once you set up the dashboard setup the sequel query. The dashboard can be automatically updated so it saves time saves people a lot of time to report program outcomes.

[Jinjing Zhao] 11:48:37
So I just want to share this feedback from one of our program directors, he has been doing program assessments for his department for like several years, I think is for like decades, we've recently created a dashboard for him.

[Jinjing Zhao] 11:48:52
So, I was just curious and asked him what his process was like before and how long I took him to do program assessments reports. And this is what he taught me like he was like a useful involved like multiple steps him begging the faculty who teaches courses

[Jinjing Zhao] 11:49:08
to get to give him the great book data.

[Jinjing Zhao] 11:49:12
This typically take two to five emails and sometimes within their offices.

[Jinjing Zhao] 11:49:17
And then later I think they, they asked instructional designers to help them like download great books from various courses and send it to him. So, imagine like on one hand, he would have got, he would have several spreadsheets off soon students assignments

[Jinjing Zhao] 11:49:33
grades right and other, on the other hand he would go to a different system and download class roster spreadsheet those as several spreadsheets, and then manually go through and find the information.

[Jinjing Zhao] 11:49:49
So yeah, we're very happy to create an automated report for him so that he can focus on like more valuable work like improving students learning where the results for short.

[Jinjing Zhao] 11:50:00
So, that is all.

[Robin Petterd - Host] 11:50:06
Questions experienced to share.

[Robin Petterd - Host] 11:50:27
Um, a couple of observations, which is interesting. So, first of all it's some people wrestle with wanting to do everything in their lm s.

[Robin Petterd - Host] 11:50:32
It seems like you've been given a lot of free. You have a lot more freedom by working with the data outside of the LM s. What would be your advice for people wanting to get started with getting their data out and do it.

[Jinjing Zhao] 11:50:46
Um, I would say the first thing you want to check us, if your institution has already ingesting the data. If not, like working with engineers to get that data is the first step, we were not able to do this with Blackboard, which is the LM s that we used

[Jinjing Zhao] 11:51:05
to use.

[Jinjing Zhao] 11:51:07
But with cameras. Apparently we get better data. so we're able to do this kind of thing.

[Robin Petterd - Host] 11:51:12
Oh, that's actually really interesting. And you were able to get it in, in an automated way as well.

[Jinjing Zhao] 11:51:19
Yes.

[Jinjing Zhao] 11:51:21
If you're interested in how to set that up. You can reach out to me. I'm not the engineer who work on data ingestion is already in our data warehouse, but I can connect you with people who know how to do this kind of thing.

[Robin Petterd - Host] 11:51:36
Yep.

[Robin Petterd - Host] 11:51:39
Find it by the tagging process really interesting for lots of reasons, big fan of tagging not actually going to do some more work with our clients and the near future about taking courses availability and now for assessment and reporting as well, so thank

[Robin Petterd - Host] 11:51:51
you.

[Robin Petterd - Host] 11:51:53
The question I've got is have your designers to factor in the tags when I designing courses, so they're actually thinking about the data from day one.

[Jinjing Zhao] 11:52:08
Wait, which one. What questions. Sorry. Sorry. It's great to have you.

[Jinjing Zhao] 11:52:15
Have you are you now finding that you're learning designers and lecturers, thinking about the data and the tags earlier in the course design process that I'm not sure about, like, ideally, their courses, and their assignments and assessments should be

[Jinjing Zhao] 11:52:34
designed in a way that's aligned with their program outcomes, but honestly I'm not sure if that's happening or, to what extent that's happening. So that's really something that the program needs to work on.

[Jinjing Zhao] 11:52:50
So, this, I feel like program assessment is really a multi party collaboration work on my hand the program. The program do a lot of does a lot of work and then they work with the university's program evaluation office to make sure their program assessment

[Jinjing Zhao] 11:53:11
plan is good, because if they if their program assessment plan is garbage, then I can give them data but the data doesn't really wouldn't really help them right.

[Jinjing Zhao] 11:53:26
So that's, that's really comes back to the quality of evidence issue.

[Robin Petterd - Host] 11:53:31
Cool. So yeah, I think that's a bit of a push cool thing that happens through here.

[Robin Petterd - Host] 11:53:37
Lisa asked, ask the question ddx API for this.

[Jinjing Zhao] 11:53:44
No. Well, partially.

[Jinjing Zhao] 11:53:46
So, again Canvas has a good process for to like for like passing you the data so it's not, I'm not super sure how it's done.

[Jinjing Zhao] 11:53:59
For especially for assignment scores quiz scores.

[Jinjing Zhao] 11:54:17
there are some challenging like that process like we're suing the process of doing engineering. Again, the process engineering that data is not very stable.

[Jinjing Zhao] 11:54:20
Cameras already like give you like really good data before rubric roll, that's, that's challenging that's, that's something we have to ingest to API calls, and

[Jinjing Zhao] 11:54:28
But yeah, but I'm saying like the data is there.

[Jinjing Zhao] 11:54:31
If you want to get it.

[Robin Petterd - Host] 11:54:34
OK, so the data is in the system. It's not always coming through with x API and even you as a data person, work with engineers to be able to get that data into the right format for working.

[Jinjing Zhao] 11:54:47
Yeah. So the infrastructure has to be there for you to do, automated reporting. Yeah.

[Robin Petterd - Host] 11:54:56
You want anyone else got any other questions.

[Jinjing Zhao] 11:55:02
I'm looking at the chat right now.

[Jinjing Zhao] 11:55:08
David you said there was always a requirement to demonstrate document where h com.

[Jinjing Zhao] 11:55:13
Yes, yes so that comes back to the outcome assessment mapping plan.

[Jinjing Zhao] 11:55:24
What are the most difficult class you're facing with setting.

[Jinjing Zhao] 11:55:27
After setting up tags.

[Jinjing Zhao] 11:55:31
I can tell you the difficulties

[Jinjing Zhao] 11:55:38
are one.

[Jinjing Zhao] 11:55:42
It's a little hard to manage attacks. Once you have a lot of courses semester after semester.

[Jinjing Zhao] 11:55:49
So,

[Jinjing Zhao] 11:55:52
because you do have, you do need someone to like manually check if the tax are in the correct places.

[Jinjing Zhao] 11:56:02
And also, the tags are not being deleted accidentally by any instructors.

[Jinjing Zhao] 11:56:08
And also another difficulties is

[Jinjing Zhao] 11:56:12
hacking rubric Rose

[Jinjing Zhao] 11:56:16
is great for getting sort of granular measure of students performance for specific learning outcomes. But, faculty, a lot of faculty don't like grading rubric.

[Jinjing Zhao] 11:56:31
Yeah.

[Robin Petterd - Host] 11:56:33
Interesting.

[Robin Petterd - Host] 11:56:34
I'm up an observation, which is the link back to some things to do with workplaces, so we've got a number of clients at the moment, who think who almost redesigning the way they're organizing the course catalog some courses around organizational values,

[Robin Petterd - Host] 11:56:51
see that this idea of tagging tagging courses is a really nice way of being able to bring things for bring things together so you can sit there and say, Okay, well, in this particular area of values with had people do this amount of completions also works

[Robin Petterd - Host] 11:57:17
really nicely with healthcare standards in Australia, and in terms of the fact that you need to report back on equivalent of the, of the program outcomes that that program outcome that's best, quite often everywhere invaded randomly and different courses

[Robin Petterd - Host] 11:57:23
and it's very hard to actually get get those reports back together so it's taking solutions are really, really lovely solution for a couple of those problems changing.

[Jinjing Zhao] 11:57:32
Yeah. So David, asked the question, each outcome is only assessed once

[Jinjing Zhao] 11:57:40
I've seen some programs. They.

[Jinjing Zhao] 11:57:46
Okay so, so for example a coming back to a computer science program for example, the, the, the program wants us to be able to not just call it but also like can be able to communicate their projects, or their products with an audience right.

[Jinjing Zhao] 11:58:05
So, but those skills.

[Jinjing Zhao] 11:58:08
Of course should be should be taught in multiple courses like when you teach Python it's a Python course, and that those two skills should be taught in this course.

[Jinjing Zhao] 11:58:17
And here you have a job, of course, and then those skills to also be holding that course, so I've seen our teachers college, use a really like tag their outcome really smartly so you have like, for example, six to competences, and these competencies are

[Jinjing Zhao] 11:58:36
being taught over over in different courses, but they would adding, they would add a tag to their outcome as like, when the, for example, when the when the outcome is first introduced.

[Jinjing Zhao] 11:58:51
So, they will have like something like outcome measure 1.1.

[Jinjing Zhao] 11:58:56
So that means the outcome is just force introduced to students so those will be in those lower level courses and then maybe like 300 level courses, this outcome the skills are being taught.

[Jinjing Zhao] 11:59:10
Again, and but it's being enhanced, so they will have something like outcome 1.1 dot ease or enhanced right and then towards like higher division courses, then you have 1.1.

[Robin Petterd - Host] 11:59:24
For example, mastered, right. So there you can see how students were forming at different stage when they are introduced, or get exposed to this outcome Oh competencies, how to answer your question.

[Jinjing Zhao] 11:59:38
Yeah.

[Robin Petterd - Host] 11:59:41
Okay.

[Robin Petterd - Host] 11:59:44
Wrap up changing thank you so much and thank you for joining quite late in your time is what time times as well, lovely to have a deep dive into the application, all data in assess assessment, really wish I think someone's actually being able to sequence

[Jinjing Zhao] 12:00:00
so big room with actually before you but then that would have made it really really like, like, like the US so thank you very, very, very much. Thank you.

[Jinjing Zhao] 12:00:08
thank you so much for, for having me here.