You've successfully subscribed to Thematic
Great! Next, complete checkout for full access to Thematic
Welcome back! You've successfully signed in.
Success! Your account is fully activated, you now have access to all content.
Success! Your billing info is updated.
Billing info update failed.

3 best practices for coding open-ended questions

Open-ended survey questions often provide the most useful insights, but if you are dealing with hundreds or thousands of answers, summarising them will give you the biggest headache. The answer lies in coding open-ended questions. This means assigning one or more categories (also called codes) to each response. But how to go about it?

Two Key Approaches to Coding Open-Ended Responses

Key Approach #1: Manual Coding

Manual coding has been invented many years ago in qualitative research. Here, a person codes each free-text response with one or more category. If you are dealing with hundreds or fewer responses, you can do it yourself, or you can hire an agency.

Read our guide on the manual coding of open-ended responses to a survey.

Key Approach #2: Automated Coding

Automated coding has become popular in recent years. This is an area of text analytics, where algorithms are used to do this task. Data scientists could use a generic NLP library, but tuning such libraries is hard, and the results are often difficult to interpret.

Solutions like Thematic use algorithms designed specifically for coding people’s responses. We recently published a white paper explaining the reliability of automated coding and ways of measuring it objectively.

3 Best Practices for Open-Ended Coding

Whether you go for the manual or automated approach, it’s a good idea to learn best practices from people who have been dealing with text for decades.

The main recommendations for open-ended coding are:

Fight Bias

Don’t start with a conceived notion of what you are hoping to find. Make sure categories or themes emerge from data rather than are pre-defined in advanced.

Provide Coverage

Make sure that the code frame covers a wide range of contrasting categories. For example, a technology company may have categories related to the UI of the product, customer service, the brand itself, stability, specific features and so on.

Ensure Consistency

If new categories emerge over time, all previously coded responses need to be re-analyzed. Keep iterating until each customer response is tagged with all themes present in the dataset.

Making Sense of Customer Comments in Surveys

How to use coding results to make sense of customer comments in surveys?

Either you will use an existing dashboard, or you create one yourself using Excel, Numbers or Tableau.

Here are the key things you need to capture:

  • What is the overall importance of each category in your code frame?
  • How do these changes depend on the segments of data?

The best chart to capture this is the old good bar chart.  Here is an example of what it could look like in a custom dashboard: 

Example of comparing themes of Promoters and Detractors
Example of comparing themes of Promoters and Detractors

And here is an example of how it could look in Excel or Numbers:


Comparing theme distribution for Promoters, Passives and Detractors using Numbers.
Comparing theme distribution for Promoters, Passives, and Detractors using Numbers.


Of course, what it looks like is not as important as whether the analysis is accurate and useful.

For more ideas, check out our article on effective Net Promoter Score analysis. The lessons will apply to other types of customer satisfaction surveys.

Keep reading

Text Analytics Software – How to unlock the drivers behind your performance

Text Analytics Software – How to unlock the drivers behind your performance

Many organisations, large or small, gather customer feedback to improve their CX efforts and ultimately their bottom line. But gathering feedback alone can’t make much of a difference. We need to analyze our feedback to discover insights that inspire us to drive action at our organisations. Enter; Text

Melanie Disse
Melanie Disse
Thematic Analysis Software: How It Works & Why You Need It (2020)

Thematic Analysis Software: How It Works & Why You Need It (2020)

Most likely, you landed in this blog because you have too much feedback to analyze. You sent out a survey or collected reviews or other form of free-text feedback. Now that you have this feedback in-hand, what do you do with it? How can you identify common themes in responses?

Alyona Medelyan PhD
Alyona Medelyan PhD
10 insider customer experience tips according to Shep Hyken

10 insider customer experience tips according to Shep Hyken

Shep Hyken knows a thing or two about customer experience. He needs no introduction in industry-wide circles, but in case you’re not familiar with his name: Shep is an award-winning customer service and customer experience speaker, a New York Times and Wall Street Journal best-selling author, A.K.A

Agi Marx
Agi Marx
Churn & Loyalty