At a Glance
- Tasks: Architect automated evaluation pipelines and design methods to assess answer quality.
- Company: Join a cutting-edge tech company focused on AI and machine learning.
- Benefits: Competitive salary, flexible work options, and opportunities for professional growth.
- Other info: Be part of a small, high-impact team driving significant product changes.
- Why this job: Make a real impact on product quality while working with innovative technologies.
- Qualifications: PhD or MS in a technical field, 4+ years in data science, strong Python and SQL skills.
The predicted salary is between 60000 - 80000 £ per year.
Requirements
- PhD or MS in a technical field or equivalent experience
- 4+ years of experience in data science or machine learning
- Strong proficiency in Python and SQL (expected to write production-grade code)
- Experience building within a modern cloud data stack, specifically AWS and Databricks
- Comfortable with agentic coding workflows and using AI-assisted development tools to iterate faster
- (Desirable) 1+ years of experience working with LLMs at scale, specifically with LLM-as-a-judge setups
- (Desirable) Prior experience working on customer-facing web products or consumer apps, with real user traffic at scale
- (Desirable) A strong research background, with experience applying research methods to real-world ML problems
- (Desirable) Experience defining evaluation metrics (e.g., factual consistency, hallucination rate, retrieval precision) and building ground truth datasets
What the job involves
- Architect and maintain automated evaluation pipelines to assess answer quality across Perplexity's products, ensuring high standards for accuracy and helpfulness
- Design evaluation sets and methods specifically to measure the impact of tool calls (particularly web search retrieval) on the final answer's quality
- Develop VLM-based solutions to programmatically evaluate how final answers render visually across different platforms and devices
- Continuously review public benchmarks and academic evaluations for their applicability to the Perplexity product, adapting and incorporating them into our regular performance measurements
- Operate within a small, high-impact team where your evaluation metrics directly shape product changes, collaborating closely with technical leadership to measure and improve Answer Quality
Data Scientist (Evals, Member of Technical Staff) employer: Perplexity AI
At Perplexity, we pride ourselves on being an exceptional employer that fosters a collaborative and innovative work culture. As a Data Scientist, you will have the opportunity to work with cutting-edge technologies in a dynamic environment, where your contributions directly influence product quality and user experience. We offer competitive benefits, a commitment to employee growth through continuous learning opportunities, and the unique advantage of being part of a small, high-impact team in a thriving tech hub.
StudySmarter Expert Advice🤫
We think this is how you could land Data Scientist (Evals, Member of Technical Staff)
✨Tip Number 1
Network like a pro! Reach out to folks in the industry, attend meetups, and connect with potential colleagues on LinkedIn. We all know that sometimes it’s not just what you know, but who you know that can land you that dream job.
✨Tip Number 2
Show off your skills! Create a portfolio showcasing your data science projects, especially those involving Python, SQL, and cloud technologies like AWS. We want to see your work in action, so make it easy for hiring managers to see what you can do.
✨Tip Number 3
Prepare for interviews by brushing up on your technical skills and understanding the latest trends in machine learning. We recommend practising coding challenges and discussing your past experiences with evaluation metrics and LLMs to impress your interviewers.
✨Tip Number 4
Don’t forget to apply through our website! It’s the best way to ensure your application gets seen by the right people. Plus, we love seeing candidates who are genuinely interested in joining our team at StudySmarter.
We think you need these skills to ace Data Scientist (Evals, Member of Technical Staff)
Some tips for your application 🫡
Tailor Your CV:Make sure your CV is tailored to the Data Scientist role. Highlight your experience in data science, machine learning, and any relevant projects you've worked on. We want to see how your skills align with what we're looking for!
Showcase Your Skills:Don’t just list your skills; demonstrate them! Include specific examples of your proficiency in Python, SQL, and any cloud technologies like AWS or Databricks. We love seeing real-world applications of your expertise.
Craft a Compelling Cover Letter:Your cover letter is your chance to shine! Use it to explain why you're passionate about this role and how your background makes you a perfect fit. We appreciate a personal touch that shows us who you are beyond your qualifications.
Apply Through Our Website:We encourage you to apply directly through our website. It’s the best way for us to receive your application and ensures you’re considered for the role. Plus, it’s super easy – just follow the prompts!
How to prepare for a job interview at Perplexity AI
✨Know Your Stuff
Make sure you brush up on your Python and SQL skills. Be ready to demonstrate your ability to write production-grade code, as this is crucial for the role. Practise coding challenges that reflect real-world scenarios you might face in the job.
✨Showcase Your Experience
Prepare to discuss your past projects, especially those involving data science or machine learning. Highlight any experience with AWS, Databricks, or LLMs, and be specific about how you contributed to the success of these projects.
✨Understand Evaluation Metrics
Familiarise yourself with evaluation metrics like factual consistency and hallucination rates. Be prepared to discuss how you've defined and used these metrics in previous roles, as they are key to the job's responsibilities.
✨Be Ready to Collaborate
This role involves working closely with a small team, so be prepared to talk about your teamwork experiences. Share examples of how you've collaborated with technical leadership to drive product improvements, as this will show you're a good fit for their high-impact environment.
