Patient Information Leaflets (PILs) educate users about medication usage, safety, and potential side effects. Ensuring these leaflets are clear, user-friendly, and accessible requires rigorous readability testing. In the EU and UK, user testing of PILs has been mandatory for 20 years, with the FDA encouraging comprehension testing of medication guides. Other regions have also introduced mandatory or recommended testing for patient information.

Two primary approaches dominate this field: Formative and Summative readability testing. Each serves distinct purposes and offers unique strengths and limitations. When strategically combined, they enhance the effectiveness, usefulness, and accessibility of PILs. By understanding these approaches, healthcare communicators and pharmaceutical companies can make informed decisions on when and how to use them.

Understanding Formative Readability User Testing

Formative testing is conducted during development to identify issues early and enable iterative improvements. It is diagnostic and improvement-focused, ensuring problems are resolved before finalisation. It is user-centric, involving cycles of testing, feedback, and revision to enhance clarity and usability. This dynamic, iterative process allows developers to make targeted adjustments, ensuring the leaflet is clear, accessible, and relevant.

The Strengths of Formative Testing

• Iterative Improvement: Ongoing revisions ensure early identification and correction of issues, reducing the risk of poor information and enhancing user satisfaction.

• Detailed Feedback: Offers valuable insights into specific problem areas such as confusing terminology or unclear instructions, allowing targeted adjustments.

• Cost-Effective: Addressing issues early reduces the expense of later revisions.

• User-Centred Design: By involving users throughout the phases of development, formative testing ensures the leaflet resonates with the audience, enhancing relevance and accessibility.

Weaknesses of Formative Testing

• Resource Intensive: Requires multiple testing rounds, participant recruitment, and feedback analysis, which can be demanding for organisations with limited budgets or tight timelines.

• Risk of Overfitting: Too much focus on feedback from a small, non-representative sample can lead to over-customisation, reducing generalisability.

• Lack of Comprehensive Evaluation: Concentrates on continuous improvement without providing a complete measure of overall effectiveness or real-world impact.

Understanding Summative Readability User Testing

Summative testing is conducted after development to evaluate overall effectiveness, usability, and user satisfaction. It is outcome-focused, providing a comprehensive assessment of whether communication goals have been met. Unlike formative testing, summative testing is a one-time evaluation, ensuring that key messages are communicated clearly and effectively.

Strengths of Summative Testing

• Comprehensive Evaluation: Assesses overall effectiveness, including comprehension, usability, and user satisfaction.

• Objective Measurement: Utilises comprehension scores and usability ratings, providing a data-driven approach to evaluate impact and effectiveness.

• Benchmarking and Validation: Enables comparisons with different leaflet versions or industry standards, useful for regulatory compliance and competitive analysis.

Weaknesses of Summative Testing

• No Iterative Improvements: Conducted after development, limiting opportunities for revisions.

• Resource Constraints: Requires larger, more representative samples for valid generalisations, increasing costs and timelines.

• Limited User Insights: Evaluates user understanding but does not explore reasons behind users' interpretations, limiting behavioural insights.

Discussion: Strategic Use of Formative and Summative Testing

The original method of diagnostic testing in document design, such as the approach pioneered by David Sless, is formative. It is meant to be user-centred. It is meant to be iterative: Test, then make changes; test again, then make more changes, and so on. This kind of approach is integrated into the FDA’s guidance for comprehension testing of Medication Guides and in other countries’ recommendations for testing patient-facing documentation. It is also the spirit in which any kind of usability testing of technologies is conducted.

The UK and EU method, however, is neither here nor there. It is intended as a formative approach, but to be conducted at the end of the leaflet design process. So, it becomes summative. The intention is to conduct tests with two sets of 10 participants; after the first set, changes to the document would be made, then the results from the second round of 10 participants would confirm the effectiveness of the changes made. Apart from the fact that the participant numbers are too high for each round of testing – a whole other discussion - hardly any changes are ever made between the two rounds. This can be due to regulatory reasons, e.g. EMA may have already approved the leaflet text before testing or because the sponsor has started the testing at a late stage and need to rush it through the process.

A common practice is to feed the leaflet through 20 participants, with user testers hardly taking a breath between rounds of testing, as if it was just one round of testing. This then becomes a checkbox activity, when it ought to be an opportunity to research the users’ perspectives. The original methodology grew its own legs and became a different animal: you can hardly call it user testing. It has just become another bureaucratic step for a product to be approved. It is more like tokenistic consumer consultation.

Formative and summative approaches do not have to be mutually exclusive. They can complement each other in the development of any patient-facing document. By strategically combining both methods, healthcare communicators and pharmaceutical companies can maximise usability, comprehension, and impact, ensuring that PILs are effective educational tools. This integrated approach bridges the gap between continuous improvement and final validation, delivering clear, accurate, and impactful patient education materials. It is a question of appetite: would companies want to spend money on this? Some do.

Thank you for reading,

Mark Gibson

Leeds, United Kingdom

References:

Best practice guidance on patient information leaflets (PILs), MHRA, UK, November 2020

GUIDELINE ON THE READABILITY OF THE LABELLING AND PACKAGE LEAFLET OF MEDICINAL PRODUCTS FOR HUMAN USE Revision 1, European Commission, 12 January 2009

Guidance for Industry: Label Comprehension Studies for Nonprescription Drug Products, August 2010

Sless D. PIL testing: Misapplied and out of context. Regulatory Rapporteur 2007; November: 14–15.

Sless D, Shrensky R. Writing about medicines for people. Sydney: Australian Self Medication Industry; 2006