Test identification

Name of test Progress in Understanding Mathematics Assessment - New PUMA primary
Version New PUMA/PUMA primary
Previous version(s) PUMA Year 1 Autumn, Spring, Summer; Year 2 Autumn, Spring, Summer; Year 3 Autumn, Spring, Summer; Year 4 Autumn, Spring, Summer; Year 5 Autumn, Spring, Summer; Year 6 Autumn, Spring, Summer; Year 7 Autumn, Spring, Summer; Year 8 Autumn, Spring, Summer; Year 9 Autumn, Spring, Summer
Subjects Maths
Summary Termly standardised assessments of a pupil’s mathematics attainment, plus a profile of mathematics skills. New PUMA is designed for whole-class use, with pupils of all abilities.

Assessment screening

Subscales Strands rather than subtests:Number,Operations,Fractions,Measures,Geometry,Statistics.(arithmetic and problem solving/reasoning included)
Authors McCarty, C., & Cooke, C.
Publisher RS Assessment from Hodder Education
Test source https://www.risingstars-uk.com/subjects/assessment/rising-stars-puma?gclid=CjwKCAiA7939BRBMEiwA-hX5Jx21p4PeOKk0oWOJYkIyFLLVBI8XlbjyVzFzrdPJtlYVJI957I53LRoCDgoQAvD_BwE
Guidelines available? Yes
Norm-referenced scores. Yes
Age range 5-11 Years
Key Stage(s) applicable to KS1, KS2
UK standardisation sample Yes
Publication date 2020
Re-norming date n/a

Eligibility

Validity measures available? Yes
Reliability measures available? Yes
Reason for exclusion from shortlist shortlisted

Evaluation and Appraisal

Additional information about what the test measures Mathematics attainment, plus a profile of mathematics skills
Are additional versions available? Note that New PUMA (Third Edition) will be published in full in 2021. Reviewed here is the interim guidance published in 2020. Termly tests (Reception: Summer; Years 1-6: Autumn, Spring, Summer). Note that standardisation of the Summer tests has been postponed due to the pandemic and data will be collected 2021.2nd Edition: Termly tests (Reception: Summer; Years 1-9: Autumn, Spring, Summer). Manual Stage 1: Reception - Year 2.Manual Stage 2: Year 3-6. Manual Stage 3: Year 7-9.Pencil and paper or online interactive (PUMA Interactive Year 1-6).(Original PUMA is now out of print).
Can subtests be administered in isolation? No
Administration group size small group, whole class
Administration duration 30-60 minutes depending on year.
Description of materials needed to administer test ManualPaper version: each pupil will require a test booklet, pencil/pen, eraser, ruler.Digital version: each pupil will require an interactive test credit and a computer/tablet.No calculators should be used.
Any special testing conditions? no

Response format

Response mode Electronic OR Paper and pencil
What device is required computer or tablet
Queston format. mixed
Progress through questions Flat

Assessor requirements

Is any prior knowledge/training/profession accreditation required for administration? Not stated (no)
Is administration scripted? Yes

Scoring

Description of materials needed to score test Paper version: Manual (mark scheme).Standardisation tables are in the manual or use MARK (my assessment and reporting kit) online analysis and reporting tool.Digital version: automatically marked.
Types and range of available scores Raw scores (by question, strand, overall),Age standardised scores (<70->130),Standardised score (cohort, not age standardised),Percentile (1->98),Mathematics age (varies depending on the test, full range currently <5;01->12:01 years note that this may expand further once standardisation from Summer tests is added),Performance indicator bands (based on national curriculum),Hodder scale (0-6+, for comparison against other measures e.g., PiRA or GAPS),National test estimated scaled score on KS2 test (85-115).
Score transformation for standard score Age standardised and year cohort standardised
Age bands used for norming Not stated
Scoring procedures computer scoring with direct entry by test taker/computer scoring with manual entry of responses from paper form/simple manual scoring key – clerical skills required
Automatised norming computerised/online

Construct Validity

Does it adequately measure literacy, mathematics or science?
Does it reflect the multidimensionality of the subject? Generic Maths
Construct validity comments (and reference for source) The new PUMA tests have high face validity, because tests are carefully alligned to the national curriculum using curriculum maps. However, the construct validity of tests has not been assessed statistically. Correlations with age are low (Pearson coefficient .30-.46). Weak correlations with age should not be taken to suggest low construct validity in this case because the tests are taken termly and matched to the curriculum. Therefore, age should not be as important as educational experiences.

Criterion Validity

Does test performance adequately correlate with later, current or past performance?
Summarise available comparisons Criterion validity will be assessed by exploring the relationship between the New PUMA tests and national test scores (KS1 and KS2 SATS) - correlations will be reported in Autumn 2021 (postponed from 2020 due to the COVID-19 pandemic and cancellation of SATS)

Reliability

Is test performance reliable?
Summarise available comparisons Reliability is reported in the manual. Internal consistency of all tests was excellent-good (cronbach's alpha .81-.95); 90% confidence band 3.47-6.39; 95% confidence band 4.20-7.75; SEM 2.1-3.87Neither temporal stability nor equivalance reliability are assessed.

Is the norm-derived population appropriate and free from bias?

Is population appropriate and free from bias? Yes
If any biases are noted in sampling, these will be indicated here. Sample sizes for the norming of the PUMA are large. However it is difficult to evaluate whether there are any biases in sampling due to limited information about sampling methodology and pupil demographics. Mean raw score by gender is provided in manual.

Sources

Sources McCarty, C., & Cooke, C. (2020). New PUMA primary: Progress in Understanding Mathematics Assessment: Key Stage One & Key Stage Two Interim Test Guidance. London, UK: RS Assessment for Hodder Education.