Developing Clinical Outcome Assessments (COA) that are accurate and reliable is vital for understanding patient experiences, health-related quality of life and how well treatments work. However, poorly design questions can introduce bias, lower comprehension and lead to inaccuracies. This article showcases common pitfalls, such as vague language, leading questions, and unbalanced response scales, while offering improved alternatives.

The following examples are based on real Clinical Outcome Assessments but have been rewritten in a way that preserves the syntax and phrasing while ensuring copyright distance.

1. Using language that is vague and ambiguous

A question like "Do you exercise regularly?" is problematic because “regularly” is ambiguous. What is the intended frequency? Daily? Weekly? Monthly? Different users will interpret it in different ways.

A better alternative would be to state the timeframe explicitly: “In the past month, how many days per week did you exercise for at least 30 minutes?” with the response choices “0, 1-2, 3-4, 5-6, 7”

2. Double-Barrelled Questions

Questions like “Do you feel tired and anxious?” combines two distinct feelings, namely ‘tiredness’ and ‘anxiety’, People might feel one but not the other. A better alternative would be to separate them:

“In the past week, how often have you felt tired?”

 “In the past week, how often have you felt anxious?”

Both should have clear response choices, such as “Never, Rarely, Sometimes, Often, Always”.

3. Leading and Biased Questions

Subtle bias often appears in COAs, such as

“How has the new treatment improved your symptoms compared to the previous one?”

This implies that the new medication is better and the respondent is nudged towards giving a positive answer.

A better alternative would be to remove the bias: “How would you rate the effectiveness of the new medication compared to the previous one?”, with balanced response choices, such as “much worse, worse, about the same, better, much better”

Response choices are sometimes skewed towards positive outcomes. For example:

“slightly worse, about the same, slightly better, moderately better, significantly better”

Or:

“No improvement, slight improvement, moderate improvement, significant improvement, major improvement.”

In both of these cases, there is no equal distribution of worsening and improvement options. So, patients are pushed towards selecting an option that indicates improvement.

4. Loaded Questions

An overtly loaded question, such as “Why did you not take your treatment?”, would be uncommon in a COA, but a gentler version of this might be: “What challenges have prevented you from taking the medication as prescribed by the study doctor?”

Even if the patient adhered to their prescription, the wording assumes that they had difficulties.

It would be better to neutralise the question as follows:

“In the past month, how often have you taken your medication as prescribed?”, with clear response options, such as “Always, Most of the Time, Sometimes, Rarely, Never”.

5. Unbalanced Response Scales

A question such as "How satisfied are you with your care? (Very satisfied, Satisfied, Somewhat satisfied)" is unbalanced because there are no neutral or negative options.

A more balanced version would be: "How satisfied are you with your care? (Very satisfied, Satisfied, Neutral, Dissatisfied, Very dissatisfied)"

6. Response Choices that Overlap

Many COAs present response options that overlap, which leads to confusion. For example: “How many days did you exercise last week? 0-2 days, 2-4 days, 4-6 days, 6-7 days”

A respondent who exercised 2, 4 or 6 days would fit into two separate categories.

A better approach would be to remove the overlap: “0-2 days, 3-4 days, 5-6 days, every day of the week”.

For pain severity, unclear distinctions between ‘mild’ and ‘moderate’ often arise, particularly in translation. This could be improved by additional explanations in brackets:

·       No fatigue

·       Mild fatigue (noticeable but does not interfere with daily activities)

·       Moderate fatigue (some impact on daily activities, but manageable)

·       Severe fatigue (significantly interferes with daily activities)

·       Extreme fatigue (cannot do daily activities)

7. Including Sensitive Questions Without Context

A question like "Have you ever been diagnosed with a mental illness?" could cause the patients to feel uncomfortable, without proper context or assurances of confidentiality. There are many examples of this. Many COA developers assume the clinical setting neutralises the stigma, but it often does not. Topics like mental health, and other conditions require careful phrasing. A better version would be: "In the past month, have you had feelings of sadness or anxiety that affected your daily life? Your answers are completely confidential."

8. Including Questions that are Irrelevant or Redundant

Questions that are blatantly irrelevant rarely appear in COAs, but items that are redundant or out-of-context do. For example, “when did you last listen to music?” may seem disconnected from a patient’s condition. There is only an implicit link between listening to music and the importance of this in a person’s therapy. This is an instance where the developer takes for granted the patient’s existing knowledge. The patient may not be aware of the link between music and clinical outcomes. A better way of phrasing this is to make the link explicit: “In the past week, have you listened to music to help manage your symptoms?”

Concluding Thoughts

Basic principles of questionnaire design inform the developer to:

·       Use precise, concise and simple language

·       Not to include ambiguous or double-barrelled questions

·       Reducing the cognitive burden on users of the COA.

So, why do these common errors persist? Possible explanations could be:

·       The developer in question once had a handle of best practice, but has since forgotten it

·       The developer comes from a clinical background and lacks formal training in survey methodology

·       The developer is not working with other experts in survey design.

Some are forced to work with outdated, pre-existing items that have never been updated.

How do these instances of poor practice escape unnoticed during psychometric testing, concept elicitation or content validity phases?

There needs to be:

·       More interdisciplinary collaboration in the design of COA between clinicians, psychometricians, as well as information design experts and patients

·       Testing phases of the COA need to include diverse populations

·       Design questions that are already culturally adaptable to prepare for the translation phase

·       Periodically assess and update older COAs

COAs should ensure that they are usable and accessible to patients who are going to be using them in clinical studies. They should not be developed with the sole aim of meeting regulatory requirements. Eliminating poor practices like those described in this article will lead to higher-quality data and better insights from patients.

Thank you for reading,

Mark Gibson

Leeds, United Kingdom, February 2025