Research Psychometrician

FlashAcademy is hiring a Research Psychometrician on a fixed term contract to do the measurement and statistical work on a twelve month, government funded research project in schools. You will audit our historical data, fix each analysis plan before the results are seen and run the final evaluation.

FlashAcademy is a UK learning and assessment platform for people learning English as an additional language, used by more than 700,000 learners in over 2,400 schools, colleges and workplaces. The project tests whether schools can identify special educational needs earlier and more fairly among pupils who are still learning English. It is research, not product development: the results are reported whatever they show, and nothing is built into the product during the project.

Research Psychometrician contract key information

Contract Fixed term to 30 September 2027, four days a week

Start December 2026, or January 2027 at the latest. The first data audit is covered by the existing team until you start.

Reports to Chief Technology Officer

Location Hybrid or remote within the UK. No school visits

Data You work with pseudonymised pupil records in a separate research environment, never with the live product

What you will do

As our Research Psychometrician, you will do the project’s analysis through four phases, about 178 working days between your start and September 2027. Every analysis plan is fixed and lodged with the funder before the results are seen and an independent measurement specialist reviews and signs your plans, threshold and evaluation.

Phase When What you will do What exists at the end

1. Audit and plan To December 2026 Audit the historical pupil records: how complete each field is and whether it means the same thing across the years covered. Count the usable sample, including pupils with two assessment sittings and two terms of app use. Write the analysis plan for historical records and lodge it before any analysis. Build the first comparison profiles of typical progress. Design the scoring for the new pupil tasks. Data audit report. Exploratory analysis plan lodged. First comparison profiles and a progress note.

2. Scoring fixed January and February 2027 Update the comparison profiles. Fix the scoring rules and the rules for choosing which pupils sit the tasks. Build the pipeline for incoming task data. Revise scoring after the speech and language therapist’s fairness review. Scoring and selection rules fixed under version control.

3. Modelling and evaluation March to July 2027 Report whether platform data improves on a simple comparison based on starting level and time in schooling. Write the analysis plan for the study in schools and lodge it before any model is built. Name the schools reserved for the final evaluation. Set the threshold at which a pupil is raised for professional review. Measure agreement between specialist reviewers. Run the single final evaluation on the reserved schools. Early findings for the second decision point. Confirmatory analysis plan lodged. Threshold record. Evaluation report covering identification rate, dependability of confidence and review burden.

4. Fairness and reporting August and September 2027 Analyse performance by home language, year group and starting level, and say where numbers are too small to conclude. Supply figures for the usability and commercial reports. Write the methods, findings and limitations of the final report. Document data and code for the archive. Fairness analysis. Final report sections. Archive documentation.

The study plans 400 to 800 pupils in at least eight schools, 100 to 150 specialist case file reviews and about 60 face-to-face assessments.

Methods you are likely to use

The project needs a different model for each kind of data and, as our Research Psychometrician, you choose and defend each one.

• Item response theory on the existing scale. Our assessment gives a reading and listening ability estimate. You use it for starting level and group progress and check that scores are linked across item banks and years.
• Growth models with measurement error. Change between two sittings is noisy for one pupil, so growth is estimated in a model that carries each sitting’s standard error.
• Learner models for app data. Lesson and word attempts need models with practice and forgetting terms, such as performance factors analysis or half life regression.
• Rasch or partial credit scoring for the new tasks. The sample suits one parameter models. You also test whether tasks behave differently by home language.
• Diagnostic accuracy. You set a threshold, report sensitivity with its interval, check calibration, and evaluate once on schools held out from model building.

Who you will work with

You do the analysis; others review it, sign it off and supply the data.

• Independent measurement specialist (external). Signs the sample size calculations and reviews your analysis plans, threshold and evaluation.
• Lead educational psychologist (external). Writes the protocol for specialist review, which gives you the reference judgements the system is tested against.
• Chief Technology Officer. Signs off the plans, the evaluation and the technical content of the final report.
• Head of Development and developers. Build the research environment and the pipelines that feed it.
• Product management. Tell you what each field meant in each version of the product.
• Research Project Manager. Runs the milestones and the evidence each one needs.
• Data Protection Officer. Reviews pseudonymisation, access and small number disclosure.

What we are looking for

You have applied measurement models to real assessment data and you fix your method before you see the answer.

• A postgraduate degree in psychometrics, statistics, quantitative psychology, educational measurement or a similar field, or equivalent experience.
• Hands on item response theory, including Rasch and partial credit models: calibration, scoring, linking scales and differential item functioning.
• Longitudinal or multilevel modelling that accounts for measurement error.
• Evaluation of a screening tool or classifier: sensitivity, calibration, threshold setting and held out evaluation.
• Reproducible analysis in R or Python, under version control.
• You have written an analysis plan before seeing results and kept to it, and you report null results plainly.
• You write clearly for teachers, funders and other non specialists.
• Experience of working with pseudonymised personal data.
• Educational or language assessment data, or computer adaptive testing.
• Learning curve or knowledge tracing models.
• Work on special educational needs, English as an additional language, or dynamic assessment.
• Subgroup and fairness analysis with small samples.
• Querying data in SQL or a document database.

#J-18808-Ljbffr