OpenIDEO is an open innovation platform. Join our global community to solve big challenges for social good. Sign Up / Login or Learn more

Use Data to Characterize User Experience

Rather than using data to measure outputs, we can use it to characterize the distribution of experiences within a system/organization.

Photo of Dan Ryan
2 1

Written by

One of the more common pieces of data used to characterize a college or university concerns class size.  We read about average class size and number of classes below 20 students, etc. The implication, usually, is that more small classes is better.

On the faculty side we might see data about academic advising: how many of my colleagues have how many advisees?

But might these statistics be misleading? Or, better, misdirecting in that they focus our attention on the wrong thing?

The thing about small classes is that they are small. Each one yields just a few student experiences of a small class.  What if we look at the same data but from the point of view of student experience?  Consider those advising numbers: rather than pondering the advising load my colleagues have, can I look at the data in terms of how many students students have to share their advisors with.

I looked at my own school's data a few semesters ago.  Advising "loads" vary with some faculty advising fewer than 10 and others 30-50 advisees and a few having even more.  To see how we might revise our quantitative perspective, suppose I have 5 advisors with 5 advisees each and 5 advisors with 25 advisees each. That would mean that in my student body of 150 students 17% have the experience of sharing an advisor with just 4 other students (and the attention bonus that comes with that) while 83% are sharing with 24 other students (and the attention deficit that comes with that).

Here's the actual data from a few years back. This is a histogram showing frequency of different sized advising loads but I've relabeled the axes so that the horizontal axis is in terms of the amount of advisor sharing advisees experience and the vertical axis is the fraction of all advisors.

Image title

Then we look at the cumulative distribution - as we move from the low load advisors (the my-advisor-has-more-time-for-me end of the distribution) to the high load advisors we count the total number of students who are advised.

Image title

What we see is that uneven advising loads is a problem not so much because it is not fair to advisors in terms of their work load, but because it may yield different qualities of student experiences in an area that's maybe quite critical in terms of overall effectiveness (see How College Works) and as part of institutional brand.

At my institution in 2011 over half of the student experience was with an advisor with 20 or fewer advisees. But about 20% have the experience of an advisor with over 40.

At a school like ours with roughly 900 undergraduates, if we had about 80 faculty FTE we could provide an advising experience in the 10-12 range, something maybe only about 10% of our students currently experience.

Class size analysis can use this logic to reach a different important observation.  Here's mock data for 100 course sections at a small college.  The horizontal axis is size of class, the vertical the frequency of each size class. The school might be proud to say that 62% of its classes are 20 or fewer students. 

Image title


Next, let's plot the cumulative frequency distribution. This time the vertical axis is the total number of student experiences at, or below, each class size  on the horizontal axis. We can see that only about 39% of all student classroom experiences are in classes 20 and below. 

Image title

Finally, we change the vertical axis to fraction of all student experiences and then count the fraction of all instructors we have teaching classes from small to large. We can see that we are using over 60% of our teaching effort to teach that 39% in below average sized classes and we deploy whopping 25% of instructional effort to produce under 10% of student experiences.

Image title

The take-aways here are two. First, we can usefully recast questions of work load fairness in terms of student experience. Second, the very common turn to tiny classes, individually supervised research projects, "creaming" top students into specialized lab sections, and so on, all of which we tend to laud as high impact practices, may draw on resources in an unintended way.  If our goal is more learning, these techniques may not always be our best investment. And this is driven by the idea of keeping an eye on the measurement of student experience rather than the measurement of institutional activity.




Specifically, please check all that apply:

  • An Individual

Tell us about your work experience:

Professor at Mills College @TheInnovationLab at Mills

What is a provocation or insight that might inspire others during this challenge?

There are more and less insightful ways to be data driven. It's worth trying to build new perspectives and perspective-shifting ideas into our data analysis. And right at the center can be the idea of measuring, tracking, and counting student experience rather than institutional output.

2 comments

Join the conversation:

Comment
Spam
Photo of Kate Rushton

Hi Dan!

This a really good example of how data could/should be used. Thank you!

What training to faculty members have on being a good advisor? Do some advisors have better links to industry than others?

View all comments