At a Glance
- Tasks: Join our team to conduct cutting-edge research on AI system capabilities and predictive evaluations.
- Company: AISI is at the forefront of AI safety and governance, shaping the future of technology.
- Benefits: Enjoy flexible working hours, generous leave, and a strong pension scheme.
- Why this job: Be part of a mission-driven team making impactful decisions in AI safety and policy.
- Qualifications: Ideal candidates have a PhD or equivalent experience in machine learning or related fields.
- Other info: This role offers mentorship from leading experts and opportunities for professional growth.
The predicted salary is between 68000 - 76000 £ per year.
The Science of Evaluations Team AISI’s Science of Evaluations team will conduct applied and foundational research focused on two areas at the core of our mission: (i) measuring existing frontier AI system capabilities and (ii) predicting the capabilities of a system before running an evaluation.
Measurement of Capabilities: The goal is to develop and apply rigorous scientific techniques for the measurement of frontier AI system capabilities, so they are accurate, robust, and useful in decision making. This is a nascent area of research which supports one of AISI’s core products: conducting tests of frontier AI systems and feeding back results, insights, and recommendations to model developers and policy makers.
The team will be an independent voice on the quality of our testing reports and the limitations of our evaluations. You will collaborate closely with researchers and engineers from the workstreams who develop and run our evaluations, getting into the details of their key strengths and weaknesses, proposing improvements, and developing techniques to get the most out of our results.
The key challenge is increasing the confidence in our claims about system capabilities, based on solid evidence and analysis. Directions we are exploring include:
- Running internal red teaming of testing exercises and adversarial collaborations with the evaluations teams, and developing “sanity checks” to ensure the claims made in our reports are as strong as possible. Example checks could include: performance as a function of context length, auditing areas with surprising model performance, checking for soft refusal performance issues, and efficient comparisons of system performance between pre-deployment and post-deployment testing.
- Running in-depth analyses of evaluations results to understand successes and failures and using these insights to create best practices for testing exercises.
- Developing our approach to uncertainty quantification and significance testing, increasing statistical power (given time and token constraints).
- Developing methods for inferring model capabilities across given domains from task or benchmark success rates, and assigning confidence levels to claims about capabilities.
Predictive Evaluations: The goal is to develop approaches to estimate the capabilities of frontier AI systems on tasks or benchmarks, before they are run. Ideally, we would be able to do this at some point early in the training process of a new model, using information about the architecture, dataset, or training compute. This research aims to provide us with advance warning of models reaching a particular level of capability, where additional safety mitigations may need to be put in place. This work is complementary to both safety cases -an AISI foundational research effort-and AISI’s general evaluations work.
This topic is currently an area of active research, and we believe it is poised to develop rapidly. We are particularly interested in developing predictive evaluations for complex, long-horizon agent tasks, since we believe this will be the most important type of evaluation as AI capabilities advance. You will help develop this field of research, both by direct technical work and via collaborations with external experts, partner organizations, and policy makers.
Across both focus areas, there will be significant scope to contribute to the overall vision and strategy of the science of evaluations team as an early hire. You’ll receive coaching from your manager and mentorship from the research directors at AISI (including Geoffrey Irving and Yarin Gal), and work closely with talented Policy / Strategy leads and Research Engineers and Research Scientists.
Responsibilities This role offers the opportunity to progress deep technical work at the frontier of AI safety and governance. Your work will include:
- Running internal red teaming of testing exercises and adversarial collaborations with the evaluations teams, and developing “sanity checks” to ensure the claims made in our reports are as strong as possible.
- Conducting in-depth analysis of evaluations methodology and results, diagnosing possible sources of uncertainty or bias, to improve our confidence in estimates of AI system capabilities.
- Improving the statistical analysis of evaluations results (e.g. model selection, hypothesis testing, significance testing, uncertainty quantification).
- Developing and implementing internal best-practices and protocols for evaluations and testing exercises.
- Staying well informed of the details and strengths and weaknesses of evaluations across domains in AISI and the state of the art in frontier AI evaluations research more broadly.
- Conducting research on predictive evaluations using the latest techniques from the published literature on AISI’s internal evaluations, as well as conducting novel research to improve these techniques.
- Writing and editing scientific reports and other materials aimed at diverse audiences, focusing on synthesizing empirical results and recommendations to key decision-makers, ensuring high standards in clarity, precision, and style.
Person Specification To set you up for success, we are looking for some of the following skills, experience and attitudes, but we are flexible in shaping the role to your background and expertise.
- Experience working within a world-leading team in machine learning or a related field (e.g. multiple first author publications at top-tier conferences).
- Strong track record of academic excellence (e.g. PhD in a technical field and/or spotlight papers at top-tier conferences).
- Comprehensive understanding of large language models (e.g. GPT-4). This includes both a broad understanding of the literature, hands-on experience with designing and running evaluations, scaling laws, fine-tuning, scaffolding, prompting.
- Broad experience in empirical research methodologies, potentially in fields outside of machine learning, and statistical analysis (T-shaped: some deep knowledge, lots of shallow knowledge, in e.g. experimental design, A/B testing, Bayesian inference, model selection, hypothesis testing, significance testing).
- Deeply care about methodological and statistical rigor, balanced with pragmatism, and willingness to get into the weeds.
- Experience with data visualization and presentation.
- Proven track record of excellent scientific writing and communication, with the ability to understand and communicate complex technical concepts for non-technical stakeholders and synthesize scientific results into compelling narratives.
- Motivated to conduct technical research with an emphasis on direct policy impact rather than exploring novel ideas.
- Have a sense of mission, urgency, and responsibility for success, demonstrating problem-solving abilities and preparedness to acquire any missing knowledge necessary to get the job done.
- Ability to work autonomously and in a self-directed way with high agency, thriving in a constantly changing environment and a steadily growing team.
- Bring your own voice and experience but also an eagerness to support your colleagues together with a willingness to do whatever is necessary for the team’s success.
Salary & Benefits We are hiring individuals at all ranges of seniority and experience within the research unit, and this advert allows you to apply for any of the roles within this range. We will discuss and calibrate with you as part of the process. The full range of salaries available is as follows:
- L4: £85,000 – £95,000
- L5: £105,000 – £115,000
- L6: £125,000 – £135,000
- L7: £145,000
The Department for Science, Innovation and Technology offers a competitive mix of benefits including:
- A culture of flexible working, such as job sharing, homeworking and compressed hours.
- Automatic enrolment into the Civil Service Pension Scheme , with an average employer contribution of 27%.
- A minimum of 25 days of paid annual leave, increasing by 1 day per year up to a maximum of 30.
- An extensive range of learning & professional development opportunities, which all staff are actively encouraged to pursue.
- Access to a range of retail, travel and lifestyle employee discounts.
- The Department operates a discretionary hybrid working policy, which provides for a combination of working hours from your place of work and from your home in the UK. The current expectation for staff is to attend the office or non-home based location for 40-60% of the time over the accounting period.
Selection Process In accordance with the Civil Service Commission rules, the following list contains all selection criteria for the interview process.
The interview process may vary candidate to candidate, however, you should expect a typical process to include some technical proficiency tests, discussions with a cross-section of our team at AISI (including non-technical staff), conversations with your workstream lead. The process will culminate in a conversation with members of the senior team here at AISI.
Candidates should expect to go through some or all of the following stages once an application has been submitted:
- Initial interview
- Technical take home test
- Second interview and review of take home test
- Third interview
- Final interview with members of the senior team
Required Experience We select based on skills and experience regarding the following areas:
- . click apply for full job details
Research Scientist - Science of Evaluations employer: AI Safety Institute
Contact Detail:
AI Safety Institute Recruiting Team
StudySmarter Expert Advice 🤫
We think this is how you could land Research Scientist - Science of Evaluations
✨Tip Number 1
Familiarize yourself with the latest research in AI evaluations and predictive modeling. Being well-versed in current methodologies will not only help you during interviews but also demonstrate your genuine interest in the field.
✨Tip Number 2
Engage with the AI community by attending relevant conferences or webinars. Networking with professionals in the field can provide insights into the latest trends and may even lead to valuable connections that could support your application.
✨Tip Number 3
Prepare to discuss specific examples of your past research experiences, particularly those related to statistical analysis and empirical methodologies. Highlighting your hands-on experience will set you apart from other candidates.
✨Tip Number 4
Showcase your ability to communicate complex technical concepts clearly. Practice explaining your research to non-technical audiences, as this skill is crucial for the role and will be assessed during the interview process.
We think you need these skills to ace Research Scientist - Science of Evaluations
Some tips for your application 🫡
Understand the Role: Make sure to thoroughly read the job description and understand the responsibilities and expectations of the Research Scientist position. Highlight your relevant experience in AI safety, statistical analysis, and scientific writing.
Tailor Your CV: Customize your CV to emphasize your academic achievements, publications, and any hands-on experience with machine learning models. Focus on your contributions to research projects that align with the goals of the Science of Evaluations team.
Craft a Compelling Cover Letter: Write a cover letter that showcases your passion for AI research and your understanding of the challenges in evaluating AI systems. Mention specific experiences that demonstrate your ability to conduct rigorous analysis and collaborate effectively with teams.
Highlight Communication Skills: Since the role involves writing and editing scientific reports for diverse audiences, emphasize your ability to communicate complex technical concepts clearly. Provide examples of how you've successfully conveyed research findings to non-technical stakeholders.
How to prepare for a job interview at AI Safety Institute
✨Showcase Your Research Experience
Be prepared to discuss your previous research projects in detail, especially those related to machine learning and AI evaluations. Highlight any publications or presentations you've made at top-tier conferences, as this will demonstrate your expertise and commitment to the field.
✨Understand the Science of Evaluations
Familiarize yourself with the current methodologies and challenges in evaluating AI systems. Be ready to discuss how you would approach measuring capabilities and conducting predictive evaluations, as well as any innovative ideas you might have for improving existing practices.
✨Demonstrate Statistical Rigor
Since the role emphasizes statistical analysis and methodological rigor, be prepared to discuss your experience with statistical techniques such as hypothesis testing, uncertainty quantification, and model selection. Provide examples of how you've applied these methods in your past work.
✨Communicate Complex Ideas Clearly
Practice explaining complex technical concepts in a way that is accessible to non-technical stakeholders. This skill is crucial for writing reports and synthesizing results, so consider preparing a few examples where you've successfully communicated intricate ideas in a straightforward manner.