Summary

Why should you care? Anytime you want to gather data from your clients, community members, heck your kids… Question design matters. It helps you get the kind of answers that you can use, vs data that isn’t really helpful, or is biased, or incomplete, or even WAY too much to handle. Pick the right tool for the job.

Video Transcript

Why Should You Care About Open-Ended vs Multiple-Choice Questions?

Maggie: So excited to be with you today talking about an extremely exciting subject in survey design. Open-ended versus multiple choice questions. Why should our viewers care?

Karyn: You should care because you want to pick the right tool for the job. Anytime you want to gather data from your clients or community members or your kids, the way you design the question matters and it helps you get the kind of answers that you can use versus data or answers that aren’t really that helpful or are biased or incomplete or too much. If you’ve got 150 people that you need to get data from, having a bunch of open ended questions is going to be, oh great, this is what I’m going to spend the rest of my life doing is reading these answers. So it’s all about designing the question for the kind of data that you were trying to get. 

Introductions

Maggie: Fantastic. So before we jump into that, who are you? Where can we find you? What’s your purpose in life?

Karyn: Okay. The first two I can do. My name is Karyn Kelbaugh and I help small businesses with customer research, particularly interviews and I help nonprofits with evaluation and other fun nerdy things and you can find me at Heykaryn.com and I will get back to you on the purpose part. Who are you and where can we find you?

Maggie: Thank you for not asking about my purpose. My name is Maggie Hodge Kwan and like you I work with nonprofits particularly around evaluation and other services as required and you can find out more about that and about me at Creativeclarity.ca.

What are the differences?

Karyn: So we’re going to try and keep this one short today because it’s not that long of a question to answer, but I’ll just go ahead and ask you, what is the main difference or what would be the purpose of an open-ended question? Choosing an open ended question over a multiple choice question.

Maggie: Well, I think you alluded to it already in your why does this matter? Um, but it is finding that right tool for the job in front of you. So, of course when you ask a multiple choice question, you are prescribing answers for your survey taker to pick from and this certainly has its place and then when you’re asking an open ended question, you’re sort of saying, hey, there could be so many perspectives or so many different responses to this question that I need to just give you this box to blurt your thoughts into because I don’t know what all of the different possibilities are. And so again, both have their place. A couple of things that stick out to me are that in survey design, you want your survey to be as easy to respond to as possible and generally asking people a lot of open ended questions turns them off, because they have to do more thinking.

Maggie: And particularly when you’re asking those big, broad questions like name all of the strengths and the things we should celebrate about the community that you live in. You’re asking for people to give some pretty deep thought and to provide some time to think through their answer. So, yeah, both have their place. If you’re asking someone a frequency question, for example, how often do you go to the gym? Multiple choice is perfect because there are only so many different options and it’s easy for someone to say, Oh, 1-3 times a week. On the flip side, if you’re asking someone to describe maybe the three biggest pain points of a current problem they’re having, it might make more sense to leave it open ended so that, their perspective and their circumstance can really come through.

Karyn: So it’s like when you’re thinking about it, if you already, if the potential answers already have some boundaries or parameters, then it might be a better choice to go ahead and pick the multiple choice because you’ve got, like you said, like with the time, if it’s an how often, when. Things that already have a known quantity and are easy answers. But things that you’re doing that are more exploratory. Like I’m just restating what you stated another way, but you know, things that you don’t know where the edges are. I think this is probably the kind of things that people would put, but I don’t know yet. Another way I like to think about it is, especially if you’re kind of in a project where you’re trying to figure something out, like your end goal might be to create a survey you could ask a lot of people. Starting with those open ended exploratory questions with a small number of people to give you your top five answers for things. Let’s find out what everybody says, oh look, all of these people gave me consistently these five things came up as options and then you’re like, okay, so now I can make a multiple choice question was an other box. Me and other boxes. I can’t let them go. I’m just like, but what if they have a different response?

Maggie: Well, best of both worlds, right? You’re finding a lot of people who will use one of the answers that you’ve provided and you’re still giving space for people who come at it from a different angle.

Pros and Cons

Karyn: Trying to think about pros and cons of each. So if I were going to say the pros and cons of open ended, pros being it’s discovery, you can ask, now, obviously you have to have a good question, but we’ll get to that later. But in your questions, you’re giving people room to write whatever they think however they want without your limitations being imposed on them so you can learn more that you didn’t even know you didn’t know. The downside is that you then have to read all of those responses and process that information and make it useful, not necessarily to quantify it, but to make sense of it. And so that’s just more work.

Maggie: Yes. So if you’re designing a survey and you need 500 responses to it. I’ve been here. I had a survey monkey survey that had a number of open ended questions and printed off that is 493 pages.

Maggie: So if you are going for a large number of respondents, yes, that’s something that you’d want to plan for in terms of time, is that even reading the responses takes time. Then adding onto that, coding them somehow making sense of them and then preparing them or presenting them in a way that is meaningful and actionable. So there is a con.

Karyn: And then for multiple choice, obviously the pro is that you get this neat and tidy dataset. You’re like, oh, okay. You know, 10 percent of people said this, 20 percent of people said this, and the analysis part is literally just okay, let’s look at my report and see what everybody said. The cons, to me are that if you have used it for something that might, there might be other answers out there that you didn’t account for, you are looking at this little narrow window of possible choices. And if you didn’t do the work upfront and you’re just saying, oh, I’ll just pick the ones I think everyone will answer, then you’re just going to get confirmed what you thought everyone would answer. People will say, well, none of those are what I would do. Bye bye. And they leave.

Karyn: Obviously, if it’s what time do you go to the gym, that one’s going to be an easy yes, but if there’s one where you had to make a decision on what choices were included, then you can be building in bias to your question so that you’re thinking, well, I was right. Those are all the choices that everyone selected. And really that’s the only ones you gave them.

Maggie: So for example, if you ask a multiple choice question that says how many minutes a day or how many times a day do you check instagram? It’s scary and people are probably confronted by the responses. But I think people could say, you know, seven times or I look at it 18 times a day or whatever it is. There is there some kind of defined range.

Maggie: But if you create a multiple choice question that says, choose which of the following values are most important from a small business that you would support. And then you list honesty, transparency, community, whatever it is, you’re making a huge assumption that someone’s most important value is even on that list. In other words, unless you give them 200 choices, which is going to be a horrible experience, you are putting your own bias in the options.

The Best of Both

Karyn: So the way to find that happy medium then is to use the open ended piece or other more discovery based or exploratory way of finding that out and talking to people and learn what the most common values are that keep coming up and then you have an educated list of choices that you can ask people and feel more confident that you’re including the ones that are the most popular.

Maggie: Yes. And throw in that other box for good measure.

Karyn: Other boxes are your friends! Then you’ll get random comments. I love surveys, they’ll answer it, but then they also are, okay, I just wanted to tell you this survey was really long.

Maggie: Yeah, I did a demographic survey once that asked people about their sexual orientation and knowing that there are many ways to define orientation, put a comment box and boy, lots of pickup lines inserted in that comment box.

Final Thoughts

Maggie: So in summary, multiple choice questions are really helpful when you are surveying lots of people, when you might have a limited amount of time to analyze and make sense of the data and when you know the subject or know the quantities really well and you can confidently provide options that meet the potential responses of the people taking your survey. And on the flip side, open ended questions are great when you don’t know and you need to do that discovery and they’re helpful when you’ve been able to make the time to really analyze them and make sense of them.