At a Glance
- Tasks: Build and maintain automated evaluation pipelines to enhance answer quality across Perplexity's products.
- Company: Join a leading tech company transforming how users access information with innovative AI solutions.
- Benefits: Competitive salary, flexible work options, and opportunities for professional growth.
- Other info: Collaborate with a high-impact team and shape product changes directly.
- Why this job: Make a real impact on user experience by improving AI-driven answers in a dynamic environment.
- Qualifications: PhD or MS in a technical field, 4+ years in data science, strong Python and SQL skills.
The predicted salary is between 60000 - 80000 £ per year.
Perplexity serves tens of millions of users daily with reliable, high-quality answers grounded in an LLM‑first search engine and our specialized data sources. We aim to use the latest models as they are released, but the intelligence frontier is a jagged one, and popular benchmarks do not effectively cover our use cases. In this role, you will build specialized evals to improve answer quality across Perplexity, covering search‑based LLM answers and other scenarios popular with our users.
Responsibilities
- Architect and maintain automated evaluation pipelines to assess answer quality across Perplexity's products, ensuring high standards for accuracy and helpfulness.
- Design evaluation sets and methods specifically to measure the impact of tool calls (particularly web search retrieval) on the final answer's quality.
- Develop VLM‑based solutions to programmatically evaluate how final answers render visually across different platforms and devices.
- Continuously review public benchmarks and academic evaluations for their applicability to the Perplexity product, adapting and incorporating them into our regular performance measurements.
- Operate within a small, high‑impact team where your evaluation metrics directly shape product changes, collaborating closely with technical leadership to measure and improve Answer Quality.
Qualifications
- PhD or MS in a technical field or equivalent experience.
- 4+ years of experience in data science or machine learning.
- Strong proficiency in Python and SQL (expected to write production‑grade code).
- Experience building within a modern cloud data stack, specifically AWS and Databricks.
- Comfortable with agentic coding workflows and using AI‑assisted development tools to iterate faster.
Preferred Qualifications
- 1+ years of experience working with LLMs at scale, specifically with LLM‑as‑a‑judge setups.
- Prior experience working on customer‑facing web products or consumer apps, with real user traffic at scale.
- A strong research background, with experience applying research methods to real‑world ML problems.
- Experience defining evaluation metrics (e.g., factual consistency, hallucination rate, retrieval precision) and building ground truth datasets.
Member of Technical Staff (Data Scientist, Evals) employer: Perplexity
Perplexity is an exceptional employer that fosters a dynamic and innovative work culture, where your contributions directly influence the quality of answers for millions of users. With a strong emphasis on employee growth, we provide opportunities to work with cutting-edge technologies in a collaborative environment, ensuring that your skills are continuously developed while making a meaningful impact in the field of data science. Located in a vibrant tech hub, our team enjoys a supportive atmosphere that encourages creativity and professional advancement.
StudySmarter Expert Advice🤫
We think this is how you could land Member of Technical Staff (Data Scientist, Evals)
✨Tip Number 1
Network like a pro! Reach out to folks in the industry, attend meetups, and connect with people on LinkedIn. You never know who might have the inside scoop on job openings or can refer you directly.
✨Tip Number 2
Show off your skills! Create a portfolio showcasing your projects, especially those related to data science and machine learning. This gives potential employers a taste of what you can do and sets you apart from the crowd.
✨Tip Number 3
Prepare for interviews by practising common questions and scenarios relevant to the role. Think about how your experience aligns with the responsibilities listed in the job description, and be ready to discuss your past work in detail.
✨Tip Number 4
Don’t forget to apply through our website! It’s the best way to ensure your application gets seen by the right people. Plus, it shows you’re genuinely interested in joining our team at Perplexity.
We think you need these skills to ace Member of Technical Staff (Data Scientist, Evals)
Some tips for your application 🫡
Tailor Your CV:Make sure your CV is tailored to the role of Member of Technical Staff (Data Scientist, Evals). Highlight your experience with data science, machine learning, and any relevant projects that showcase your skills in Python and SQL.
Craft a Compelling Cover Letter:Your cover letter is your chance to shine! Use it to explain why you're passionate about improving answer quality and how your background aligns with our mission at Perplexity. Be specific about your experience with LLMs and evaluation metrics.
Showcase Your Projects:If you've worked on any relevant projects, especially those involving automated evaluation pipelines or cloud data stacks, make sure to mention them. We love seeing practical applications of your skills!
Apply Through Our Website:Don't forget to apply through our website! It’s the best way for us to receive your application and ensures you’re considered for the role. Plus, it shows you’re keen on joining our team!
How to prepare for a job interview at Perplexity
✨Know Your Stuff
Make sure you brush up on your data science and machine learning knowledge, especially around LLMs. Be ready to discuss your experience with Python, SQL, and any cloud platforms like AWS or Databricks. They’ll want to see that you can write production-grade code, so be prepared to showcase your technical skills.
✨Showcase Your Projects
Bring examples of your past work, particularly any projects where you've built evaluation metrics or worked with automated pipelines. Discuss how your contributions improved answer quality or user experience. This will demonstrate your hands-on experience and problem-solving abilities.
✨Understand the Company’s Needs
Familiarise yourself with Perplexity's products and their approach to answer quality. Think about how your skills can directly impact their goals. Being able to articulate how you can contribute to their mission will set you apart from other candidates.
✨Ask Insightful Questions
Prepare thoughtful questions about the team dynamics, the challenges they face in improving answer quality, and how they measure success. This shows your genuine interest in the role and helps you gauge if it’s the right fit for you.
