Assessment Rocks

Not-so-helpful Advice

I solemnly swear that this page contains No Actual Useful Advice. If you want to learn how to do assessment effectively, check out the resources page, attend a conference, or try reading a book. If you want to have a little fun, continue below. New articles posted occasionally. This page may include paid promotions or affiliate links. You can also just send me cash through the mail.

Assessment: This is the Way logo in the style of the Mandalorian

In the Disney+ TV series The Mandalorian, The Mandalorian (named Din Djarin) lives his life by a set of very strict creeds. Three of the most commonly used are: never remove your helmet in front of another living being, always help another Mandalorian, and never finish a sentence with a preposition.

Mandalorian and Grogu

He follows these three creeds extremely carefully throughout the first two seasons. Amazingly, even though he spends much of his time on desert planets and in various spaceships (with no obviously visible humidifiers), he never once removes his helment to pick his nose. Frequently, when a decision is made that aligns with some aspect of the Mandalorian creed, Din (or other Mandalorians) state "This Is the Way." This statement immediately ends any additional discussion about what to do, strategy, etc. However...in the last episode of season 2, he meets Bo-Katan Kryze, the leader of a group of Mandalorians known as the Mandalore Resistance. Bo-Katan and her followers have also pledged to follow the creed of the Mandalorians, but shockingly they are more than happy to remove their helmets. THIS IS NOT THE WAY Mando (as his friends call him) must be screaming in his head. He his SHOCKED. Simply SHOCKED! To see that others who claim to follow the creed behave so differently than him (I'm still waiting to see if, when he finally removes his helment, he goes straight for the nose.)

What does this have to do with assessment? Well, there are many similarities between us assessment leaders and the Mandalorians. We too have taken a form of a creed, in that we have committed our work to the task of ensuring and improving student learning. And, also like the Mandalorians, too often we also dogmatically follow elements of the "creed" that, while perhaps made sense at one point, are not critically important to our work.

For the Mandalorians, maybe it made sense at one point to always wear helments. Maybe rocks were falling from the sky and members kept getting bonked in the head. Maybe some of the early Mandalorians were unbearably ugly. And by the start of season 3 it is obvious that, whaterver the reason once was, it is now lost to time.

Similarly, as assessment leaders we may have committed ourselves to a specific piece of assessment dogma: maybe it's insisting everything be measured with a rubric, or maybe it's requiring every single program, no matter how small or large, to complete a report annually using the same 15-page template, or maybe it's insisting that only certain action verbs can be used to write effective learning outcomes statements...those are all dogma...and what is most important is that we keep focused on why we got into this work in the first place: to ensure and improve student learning. And let everything else go...

Unsatisfied with Satisfaction

Interpreting results from satisfaction surveys can be a bit tricky. The most important thing to remember about satisfaction is that it is the gap between expectations and performance. If performance exceeds expectations, then you have positive satisfaction. If performance is below expectations, then you have dissatisfaction.

Let me provide an illustration. I was on a car trip across the state to give a presentation at an institution. The drive was 4.5 hours, and by the time I arrived in town I was looking forward to getting a fresh, crispy, juicy salad for dinner (I know a salad isn't rock-and-roll, so don't give away my secret.) I had in mind a salad perhaps a bit like this:

A very Tasty Salad

I completed my order at the restaurant, drove back to the hotel, went up the stairs to my room, and sat down in front of the TV to watch the football game and enjoy my tasty, juicy salad. I opened the clamshell, and this is what I saw (this is the real, actual picture of my salad):

a horrible salad

I laughed out loud. I laughed so hard it totally made the $4.99 I paid for the salad totally worth it. I've sat through comedy movies that cost me $8.50 plus $12 for popcorn and soda where I didn't laugh nearly as hard as I laughed at this salad. In my mind I was imagining the conversation in the kitchen:

Manager: "Someone finally ordered a salad! I'll need it to go in five minutes!"

Cook (a 16-year-old who doesn't really want to be at work in the first place): "It's a Sunday night! We are running low on a few items! Like: carrots, tomatoes, onions, cheese, cucumbers, olives, lettuce."

Manager (who thought he would be running an upscale, four-star bistro instead of this dumpy diner): "I SAID, give me a salad! Now you are down to 3 minutes!"

Cook: "Fine. All we have is some brown lettuce that I found under the ketchup in the back of the fridge. I was going to eat it to give myself food poisoning, but I suppose I could give it to this idiot."

Now, that's dissatisfaction!

GOAT Coffee

I heard on the news yesterday that one of the greatest metal guitarists of all time - Kirk Hammett of Metallica - is now selling his own coffee line. Hearing this news made me feel really old. And not because coffee is a drink I always considered a drink for grown-ups. And then I felt old.
Coffee on the top of a mountain with cozy chairs and a table

Then after poking around on the internet a bit, I realized I am probably the only person left in the United States who does not already have their own coffee line. Making matters worse, this means that almost every good coffee company name is now taken. For example, there's a coffee company called "GOAT" (Greatest Of All Time). Think about that for a moment. They are claiming to the produce The Greatest Coffee Of All Time. Not just good. Not great. Not excellent. The Greatest Of All Time. While as a rock-and-roller, I appreciate the idea of going big or going home...I wonder if it is even possible to defend this assertion. I mean, how many coffees can they have tested in their lifetime? What evidence do they have that they really are better than any other coffee? Their website includes "their story," which says its mission is to "craft new coffee experiences for you." I am hard pressed to describe a "coffee experience," let alone a New Coffee Experiences. Most of the time, I don't want an experience - I just want something to keep my head off my keyboard at 2:00 PM.

In my experience, most higher education institutions have at least one GOAT departments. (If you work at a school with an animal science program your school might literally have a GOAT department.) By GOAT, I mean departments that are so full of themselves, so obsessed with their own greatness and excellence.

How to spot a GOAT department? Here are some key symptoms:

GOAT departments. GOAT coffee. The lesson from assessment is that there is no such thing. That no matter how good you are, you can always be better. And the moment you stop working to improve, the moment you think you've made it, is the moment when you start to decline. Think about it: Michael Jordan, the greatest to ever play the game, practiced exteremely hard. He was the greatest of all time and yet he still put in the work. Back at it (but maybe a refresh on my coffee first!).

Never in the History of Humankind

There's an old saying that if you put 1,000,0000 monkeys on 1,000,0000 typewriters for 1,000,000 years, eventually one of the monkeys will fling poo directly into your face and laugh. I'm not sure about the monkeys but I know my kids would laugh.

But the point of the monkey story is that, given enough time and opportunities, any event can occur by chance. I call BS (baloney stuff) on this one. Here's my counter-argument:

In the course of my career in higher education I have often been responsible for presenting program enrollment numbers to program leaders and faculty. It seems like a simple request: simply count up how many students are enrolled in a certain degree program. But it doesn't matter if the counts I am presenting are for undergraduate programs or graduate programs, large or small programs, new programs or old programs, or programs whose name is an anagram for "adieu crotch Teena", invariabley upon receiving the reported counts, a member of the faculty committee will, after frowning at the numbers for a few minutes, state "I don't think this is quite correct."

It is possible the lack of agreement between the count of students in the program I produce and what faculty expect to see is due to some systematic error I cause. As any reader of this site can tell you, I am by no means perfect. I've evn ben known to mke a spleling eror or too.

It is also possible that the discrepancy is due to the complexity of how students enroll in our programs. Some students enroll in more than one program (should we count them in one program to avoid double counting or both to capture their true enrollment pattern), some students enroll part-time (are you interested in enrollment FTE, to gauge the load on the faculty or headcount, in case you want to stuff them all in a room to tell them the program is moving online immediately due to the outbreak of an infectious disease), and sometimes students participate in programs before they are fully enrolled.

But I think the most likely reason, and you can quote me on this, is lack of sufficient funding. Due to the pandemic, budget cuts, and a worldwide banana shortage, for the last two years my office has been getting by with only 734,852 monkeys typing on 734,852 typewriters. But with proper funding we can get my monkey team back above that 1,000,000 mark and then, some day in the future, I will have the joy of sitting in a meeting and waiting for that magical moment when a data user carefully reviews program enrollment counts, looks up with a smile slowly spreading across their face, and says, "By jove, I think you've done it!"

Let's Face it: Nobody is Reading your Report - Not Even on Accident

"5:00 PM: Solve world hunger: tell no one."

One of the top fallacies in assessment - in all of science, really - is that a well-writte report, posted on your lousy website, with the answer to an immediately pressing problem - like solving the puzzle of why there is such a high correlation between the amount of blood running out of your nose and the likelihood that the kleenex box will be empty (just ask Billie Eilish) - will be sufficient to create action to solve your problem. (Here's a preview: the solution? Stop picking your nose.)

Unfortunately, a study by the World Bank found few of their PDF reports received much attention. In fact, just under a third had never been downloaded, not even once, and only 13% had more than 250 downloads. Those are pretty sad statistics, especially because almost every website now gets dozens of hits from random web bots who want to steal your content (I solved this by not having any content of any value - haha, take that!).

But maybe, if you are like me, you are thinking, "but I am sure my assessment reports are far more interesting than some stupid World Bank report on world hunger and economic policy!" Now that I've written this down I can see how ridiculously insane that sounds. Because I hope avoiding worldwide economic financial meltdowns and decreasing world hunger is more important than discovering Jackson Hall's floor 8's level of satisfaction with our recent response to their bedbug infestation (so you don't have to go looking: 43% were "screaming into their pillows").

It is therefore obvious that if we want people to pay attention to our assessment and scientific findings, we are going to have to find other ways of communicating these results. Creating a PDF, no matter how beautifully prepared, and dropping it on an obscure page on your lousy website on a Friday afternoon is not getting it done. Hey, here's an idea! Collect a bunch of empty kleenex boxes (these should be easy to find the next time you have a bloody nose). Then fill them with copies of your latest report, and place them back in their various locations. Turn off your building's humidifier, stir up some dust into the air, and, soon enough, people all over your building will be putting their eyes on your report. Problem solved!