Test identification

Name of test Progress in Understanding Mathematics Assessment (KS3)
Version PUMA KS3
Previous version(s) PUMA Year 1 Autumn, Spring, Summer; Year 2 Autumn, Spring, Summer; Year 3 Autumn, Spring, Summer; Year 4 Autumn, Spring, Summer; Year 5 Autumn, Spring, Summer; Year 6 Autumn, Spring, Summer; Year 7 Autumn, Spring, Summer; Year 8 Autumn, Spring, Summer; Year 9 Autumn, Spring, Summer
Subjects Maths
Summary A suite of standardised, paper-based termly tests to measure and predict student progress in maths and compare their performance against national averages.

Assessment screening

Subscales Strands rather than subtests:Number,Algebra (from Summer Year 7),Ratio, proportion and rates of change,Geometry and measures,Probability (from Summer Year 7),Statistics
Authors Burdett, N., Collett, M., Dixon, T., Nicholson, J., & Watson, C.
Publisher RS Assessment from Hodder Education
Test source https://www.hoddereducation.co.uk/subjects/assessment/assessment-series/puma-key-stage-3
Guidelines available? Yes
Norm-referenced scores. Yes
Age range 11-14 Years
Key Stage(s) applicable to KS3
UK standardisation sample Yes
Publication date 2016
Re-norming date n/a

Eligibility

Validity measures available? Yes
Reliability measures available? Yes
Reason for exclusion from shortlist shortlisted

Evaluation and Appraisal

Additional information about what the test measures Mathematics attainment, plus a profile of mathematics skills
Are additional versions available? 2nd Edition: Termly tests (Reception: Summer; Years 1-9: Autumn, Spring, Summer). Manual Stage 1: Reception - Year 2.Manual Stage 2: Year 3-6. Manual Stage 3: Year 7-9.Pencil and paper or online interactive (PUMA Interactive Year 1-6).(Original PUMA is now out of print).Note that New PUMA (Third Edition) will be published in full in 2021. Interim guidance is currently available.
Can subtests be administered in isolation? No
Administration group size small group, whole class
Administration duration 60 minutes (Section A: non-calculator 30 mins, Section B: Calculator 30 mins)
Description of materials needed to administer test ManualPaper version: each pupil will require a test booklet, pencil, pen, eraser, ruler and paper for additional rough working.No calculators should be used for part A (but can be used for part B).
Any special testing conditions? no

Response format

Response mode Electronic OR Paper and pencil
What device is required computer or tablet
Queston format. mixed
Progress through questions Flat

Assessor requirements

Is any prior knowledge/training/profession accreditation required for administration? Not stated (no)
Is administration scripted? Yes

Scoring

Description of materials needed to score test Paper version: Manual (mark scheme).Standardisation tables are in the manual or use MARK (my assessment and reporting kit) online analysis and reporting tool.
Types and range of available scores Raw scores (by question, strand, overall),Age standardised scores (<70->130),Standardised score (year cohort, not age standardised),Percentile (1->98),Mathematics age (varies depending on the test, full range <10;4->15;7),Performance indicator bands (based on national curriculum giving relative attainment term on term),Hodder scale (for comparison against other measures e.g., PiRA or GAPS),
Score transformation for standard score Age standardised and year cohort standardised
Age bands used for norming 1 month
Scoring procedures computer scoring with manual entry of responses from paper form/simple manual scoring key – clerical skills required
Automatised norming computerised/online

Construct Validity

Does it adequately measure literacy, mathematics or science?
Does it reflect the multidimensionality of the subject? Generic Maths
Construct validity comments (and reference for source) PUMA tests have high face validity, because tests are carefully alligned to the national curriculum using curriculum maps. However, the construct validity of tests has not been assessed statistically. Correlations with age are highly variable (Pearson coefficient .20-.79). Weak correlations with age should not be taken to suggest low construct validity in this case because the tests are taken termly and matched to the curriculum. Therefore, age should not be as important as educational experiences.

Criterion Validity

Does test performance adequately correlate with later, current or past performance?
Summarise available comparisons n/a

Reliability

Is test performance reliable?
Summarise available comparisons Reliability is reported in the manual. Internal consistency of all tests was excellent (cronbach's alpha .93-.95; 90% confidence interval 6.2-6.3; SEM 3.55-3.94).Neither temporal stability nor test equivalence were assessed.

Is the norm-derived population appropriate and free from bias?

Is population appropriate and free from bias? Yes
If any biases are noted in sampling, these will be indicated here. Sample sizes for the norming of the PUMA are large. However it is difficult to evaluate whether there are any biases in sampling due to limited information about sampling methodology and pupil demographics. Mean raw score by gender is provided in manual.

Sources

Sources Burdett, N., Collett, M., Dixon, T., Nicholson, J., & Watson, C. (2018). Progress in Understanding Mathematics Assessment. Manual Stage 3: Years 7-9. London, UK: RS Assessment from Hodder Education.