This workshop divides into two parts. The first establishes the importance and relevance of validation theory and practice. The second unpacks the complexities and challenges of conducting a Classroom-Based Formative Assessment (CBFA) validation exercise.

Part One will consist of an introductory overview followed by sessions focused on validation as scientific inquiry; validation as an evolving process; current conceptions of validation practice; validation frameworks; and assessment validation as evidentiary arguments and evidential reasoning. Part One will culminate in a conversation around innovations in current validation practice.

Part Two will begin with a discussion of contemporary views of classroom assessment. CBFA will then be deconstructed in terms of instrument; process; and function. Next, validity and CBFA claims will be explored against the backdrop of an argument-based approach to validation. The sufficiency and relevance of CBFA validity evidence will then be investigated. This part of the workshop will constitute the focus of discussion activities, specifically, what counts as CBFA validity evidence. Consideration will be given to how little CBFA validation evidence can be relied upon by teachers? Should be relied upon by teachers? Is relied upon by teachers? Moreover, if teachers fail to provide certain strands of CBFA evidence, would this prevent them from claiming that their classroom assessment is valid?

The workshop will provide practical guidance for the construction and validation of valid, reliable, and fair CBFAs.

By sharing experiences through a collaborative workshop environment, greater insights will be drawn leading to an increased understanding of the validation process and how it might be routinely undertaken in classroom contexts, and for differing CBFAs. Participants should have good access to internet connection, a computer with a working microphone in order to join this session.

About Stuart Shaw
Stuart is Honorary Professor of University College London (UCL) in the Institute of Education – Curriculum, Pedagogy & Assessment. He has worked for international awarding organisations for over 20 years and is particularly interested in demonstrating how validity, reliability, and fairness (in educational, psychological, and vocational assessments) is achieved. Recent books include: Validity in Educational and Psychological Assessment (Newton & Shaw, 2014); Is Assessment Fair? (Isabel Nisbet & Stuart Shaw, 2020). His latest book with Isabel Nisbet (published in 2024 by Routledge) is: Educational Assessment in a Changing World: Lessons Learned and the Path Ahead. Stuart is a Fellow of the Chartered Institute of Educational Assessors and a Fellow of the Association for Educational Assessment in Europe

The workshop starts with an introduction to the process and workflow of assembling different exam variants. This is followed by an explanation of how to analyse exam data in the Classical Test Theory (CTT). Using a same sample dataset, we will illustrate how item analyses, distractor analyses and reliability analyses can be performed.

Then the participants will practice conducting CTT analyses in small groups, using the sample dataset. Templates and instructions for the exercises will be handed out. First, the participants will analyse the data on the item level. Subsequently, they will analyse the data on the test level. The exercises will be finished with a joint discussion and recap.

After a break, the workshop continues with an introduction to the core and central concepts of the Item Response Theory (IRT). Central to the workshop is the Rasch (1PL) model. In addition, the 2PL and 3PL model are briefly addressed. Using once again a sample dataset, the model fit, person fit and item fit analyses are carried out. The results are presented and explained in a user friendly and illustrative way.

The participants will then practice interpreting the IRT outcomes in small groups. The corresponding examples and assignments will be handed out. The exercises will again be concluded with a joint discussion and recapitulation.

Finally, using a straightforward example, the trainer will present to the participants how the analysis results can be used to assemble, in an automated way, an accurate and reliable exam variant that suits the target population and measurement purpose of the exam. To conclude, the trainer will give some recommendations for further learning and for further implementation of the offered methods in practice.

NOTE: There is no preparation required for this session and is suitable for novices and experts in the assessment development field, who are seeking to improve their practice.

About Arnold Brouwer

Dr Arnold J Brouwer is an international assessment expert in educational measurement and psychological testing. He is an all round researcher, innovator and consultant of methods and techniques for analysing, optomising and auditing reliable and valid assessments. He is director of the Dutch Research Centre for Examination and Certification. Besides this, he is a guest researcher at the Department of Cognition, Data and Education within the Faculty of Behavioural Management and Social Sciences at the University of Twente. He obtained his PhD on the Systems-oriented Talent Management (STM) model, which he elaborated into a method for aligning human talent with the business purpose from both a psychological and managerial perspective.

About RCEC

RCEC, Research Center for Examination and Certification if the leading knowledge and networking organisation for assuring and advancing the quality of examinations.

As a certification authority, we evaluate the quality of examinations and award them with RCEC certification. We are involved in large national and international examination systems, both in formal education under the heading of the ministries of education and the non-formal education under the heading of autonomous administrative authorities set up by the ministries of education, as one example.

As an expertise centre, we fulfil independent and impartial counter function. We do not develop out our educational tests; we exclusively support institutions in the formal, non-formal and commercial education sectors with research and innovation. We facilitate all steps of the testing cycle: from construction to administration and from scoring to evaluation.

From our state-of-the-art academy, we provide tailor-made training and development courses and contribute to knowledge sharing through publications and presentations.

We work with passion and dedication towards our core mission: doing justice to decision by, about and for talent.

The rising use of commercial digital assessments, surveys and EdTech tools in classrooms worldwide is creating new opportunities for research and insights. If unlocked, the growing banks of data can enhance approaches to longitudinal studies, monitoring, evaluation and studying of trends at national and international levels.

This workshop will share practical experience and examples from a recent case study from the Nuffield Foundation funded research project, to look at primary school wellbeing and attainment in the wake of the covid-19 related school closures. It will reflect on learnings gathered over 5+ years of research. It will highlight the opportunity for greater insight through collaboration and allow opportunities to discuss the associated risks and ethics, drawing on experience of University and commercial partners. It will explore the landscape and offer chances for a roundtable to share opportunities in other international contexts. Covering dissemination and impact examples, it will end with a look to the future with time to co-create a shared vision.

This workshop is designed for policymakers, educational researchers, university or school practitioners, assessment developers who are interested in learning about using large datasets for research in other contexts to their own.

About Katie Blainey

As Product Director: Assessment, Professional Development and Learning Design for Hodder Education Katie has developed and delivered a large programme of bespoke, standardised tests and surveys for the K-12 market for 10 years.

Her company’s products are used widely in thousands of schools in the UK and globally to support teachers in their decision making and to improve pupils’ learning.

She leads a team with extensive digital experience of developing assessments, platforms and software to solve teacher’s problems with a strong data-rich underpinning and is proud to have won awards for SNAP (a special educational needs digital tool).

She has been Principal Investigator on the Nuffield Foundation research project to explore the impact of Covid-19 related school closures on children’s attainment and wellbeing. Overseeing the publishing of 10 white papers that have gained national press coverage.

About Hodder Education

Hodder Education is more than a global creator of quality content, product and services. For over 150 years we have worked closely with educators, academics, awarding bodies, partners and learners to design and deliver focused, accessible, flexible, pedagogically robust solutions to help teachers to teach and students to learn, wherever they are in the world. We are the second-largest educational publisher in the UK and are proud to be the market leader in Secondary and FE. We provide more educational content and services than any other UK publisher, encompassing teaching-and-learning materials, assessment resources and professional development programmes across the breadth of UK and international curricula. Our Brand Values put teachers and learners at the heart of everything we do; Quality, Trust; Collaboration.

Contextual data play a crucial role in drawing meaningful policy insights from educational assessments, such as for the comparison of sub-groups or the identification of performance predictors, moderators, or mediators. This workshop will review crucial research-based design principles that can help researchers and practitioners design better surveys at the student, teacher, or school level while avoiding some of the most common mistakes in developing surveys.

After a theoretical introduction and demonstration with examples from national and international large-scale assessments, participants will apply the design principles to a practical scenario to test and expand their own understanding.

This workshop is designed to provide valuable insights for researchers, practitioners, and policymakers with interest in improving existing large-scale assessments or setting up new large-scale assessments in their countries from scratch. While some prior knowledge in large-scale assessment or survey methodology would be helpful, it is not required.

About Dr Jonas Bertling

Jonas Bertling, Ph.D., is a Director in ETS’s Research Services Division where he provides leadership for large-scale assessment projects for national and international clients. One of Jonas’s areas of expertise is the design of contextual questionnaires for students, teachers, and school administrators. Dr. Bertling has 10+ years of experiences in leading questionnaire development for the National Assessment of Educational Progress (NAEP) in the U.S. and the Programme for International Student Assessment (PISA) internationally. A focus of his work has been establishing more robust survey design principles for these programs that maximize their policy value of contextualizing achievement results from digital assessments in cross-national contexts.

About Education Testing Services

Educational Testing Service (ETS) is a non-for-profit organization with headquarters in Princeton, NJ, United States that is committed to improving education worldwide and helping learners get equitable access to high quality education. In addition to being an industry leader in English language assessments and graduate admission testing, ETS has been a leading organization in designing, developing and administering assessments that are central to informing policy for educational transformation. These assessments include US’s National Assessment of Educational Progress (NAEP), and international assessments like the Programme of International Student Assessment (PISA) and the Programme for International Assessment of Adult Competencies (PIAAC).