Coffee, LEGO movies, questionnaires, and complexity theory

I recently had a cup of coffee with a friend, and the discussion turned to the difference between complicated and complex, and why the difference is important.

I have had reason to think about that recently, so I had a couple of examples fresh in my mind, both relating to questionnaires and surveys.

As it turns out, many questionnaires you are asked to fill out have a common design mistake: The assumption that the subject under investigation is complicated, rather than complex. It is an easy mistake to make. The result is increased risk that the survey points you in the wrong direction.

Let's briefly define what we are talking about before digging in to the meat of the matter:

• Complicated systems have many parts, but they also have predictable cause and effect relationships. For example, a mechanical watch is complicated. It is also predictable. It runs like...well, it runs like clockwork.

• Complex systems have parts that can adapt to the behavior of other parts in the system. A family is a complex system. All family members both react and adapt to what other family members do. Business organisations, countries, teams and workgroups, ecological systems, the scouts, my photo club, and aquariums are also complex systems.

You can pick a mechanical watch apart and study each piece, that is analyse it, to figure out how it works.

On the other hand, studying each member of a family, or a software development team, or each fish in an aquarium, will not necessarily tell you how the system as a whole will work. A family has emergent properties, properties that belong to the family as a whole, but not to any of its members.

LEGO movies and unpredictability

Here is an emergent property of the system consisting of my eight year old son and me: Stop motion movie making.

My son asked me if we could make a LEGO movie. I said yes, of course, and we created the short movie above. It would be impossible to predict in advance that my son and I would produce a short LEGO movie featuring Thor and The Hulk.

In retrospect, it does not seem farfetched at all that we would do such a thing. It would be easy to construct a Future Reality Tree explaining why and how we did it. However, the tree would have to be created afterwards. It would be impossible to construct a tree that accurately predicts what we will do.

What on Earth does this have to do with surveys and questionnaires? As it turns out, a lot!

Questionnaires: The art of asking the wrong questions

To find out something about a complicated system, you can ask a question about a part of the system. If you want to know more, you can continue to ask questions about parts of the system. Eventually, you can compile the answers, and they will tell you a lot about the system as a whole.

With a complex system, that does not work very well. Knowing each part won't tell you the emergent properties of the system. Another problem is that with a complex system, you do not necessarily know which parts and properties of those parts, that are important to the functioning of the whole.

Systems where humans interact, are complex systems, but questionnaires are very often designed with the implicit assumption they are complicated, or even simple. Thus, most questionnaires, even the ones you pay specialists to create, are designed wrong. They do not tell you what you need to know!

As the illustration above shows, asking many specific questions means you get specific knowledge of the things you assume are important. however, you have no real basis for making these assumptions, because you haven't studied the system yet.

For example, some time ago a coffee shop I sometimes visit made a survey using touch screen computers and a set of specific questions. The questions were about the quality of service at the counter, whether the personnel behavied in an appropriate manner, whether there were cups and plates left on unoccupied tables in sight of the computer, and other things the management wanted to know.

As a fairly frequent guest, I noted that all of the things the questionnaire had questions about worked very well. There were problems, but the questionnaire did not mention them.

For example, there were several electrical outlets that were damaged well beyond the point of being dangerous, the toilets often ran out of soap and toilet paper, the free WiFi-system did not work. While  there was no problem with cups and plates lying around near the computer terminal, there were often several tables on the second floor that could not be used because they were covered in cups, plates, and glasses.

Nobody asked about it, because they did not know they should. The things they did ask about were the things they already had control of.

Thus, the questionnaire was all but useless as a tool for improving the café. All it could do was confirm that things the management had focused on in the past were ok.

The questionnaire created a false sense of having everything under control, which reduced the incitament to do real improvement.

Query Fatigue

Thinking requires a lot of energy, and the human brain has very limited energy reserves. This means a questionnaire with many questions will tire the brains answering the questions. Thus, the quality of the replies will degrade significantly, so that replies to question 26 will be much less trustworthy than replies to question 3.

Many people will of course opt out of replying to a questionnaire altogether, if it has too many questions.

To know more, you must ask less!

So, if a questionnaire gets worse the more comprehensive it is, what can you do about it?

Well, if asking more questions makes the questionnaire worse, then you can make it better by asking fewer questions.

If you ask only a few questions, then obviously you must ask the right ones, or you will learn little of significance. You cannot know what is important to ask, but there are people who know: The people answering the questionnaire. They probably do not know it as individuals, but collectively they do.

How can you tap into that knowledge? You can ask broad questions.

If you ask broad, open-ended questions like:

• What is the single most important thing about X that we should improve?

• What are the most important problems with X?

Then you will get the respondents to tell you what they believe is important, rather than telling you their beliefs about what you believe is important to them.

The difference is quite important.

There are many ways of doing this. Personally, I like the Crawford Slip brainstorming method. I also use a modified form of Net Promoter Score. (I had to modify it, because the original version of NPS botches the statistics and makes the assumption all systems are complicated.)

Scope and analysis paralysis

There are two important questionnaire design problems I am saving for another day and another post:

• Scope: Which people should you ask? This is sometimes obvious. On the other hand, the obvious answer is often wrong, so you need to give this some thought.

• Analysis paralysis: What do you do once you get the responses to your questionnaire? How do you know which answers are important, and which ones are not?

With those two questions I bid you farewell, for now. Until next time, think about questionnaires and surveys in your own organisation. Did they really tell you what you need to know? Were the big problems solved? If not, what can you do about it?