About the job
Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel , Adam D'Angelo , Larry Summers , and Jack Dorsey .
Position: Generalist Annotator — Health AI Conversation Quality Evaluation
Type: Contract
Compensation: $20–$160/hour
Location: Remote
Role Responsibilities
- Evaluate the relevance and clarity of AI responses . Ensure the AI answers the question in understandable language for non-experts.
- Detect sycophancy in AI interactions. Identify when the AI tells users what they want to hear instead of what they need.
- Assess the tone and outcome of conversations. Determine if the user leaves informed and calm or anxious and uninformed.
- Provide written justification for ratings below the top option. Explain what went wrong and suggest improvements.
- Work independently to review conversations ranging from 1 to 7 turns. Spend 2–13 minutes per conversation based on length.
Qualifications
Must-Have
- Strong written English skills.
- Ability to explain judgments in two to four specific sentences.
- Careful reading to notice omissions in responses.
- Comfort with applying a detailed rubric consistently.
- Reliable availability for time-boxed batches.
Preferred
- Experience in annotation, human-feedback, evaluation, or content review.
- Background in explaining technical concepts to non-experts.
- Experience with rubric-based grading or quality assurance.
Important
- Using an AI tool to write comments is prohibited. This work captures human judgment that AI lacks. AI-written feedback corrupts the dataset and is checked for.
Resources & Support
- For details about the interview process and platform information, please check:
- For any help or support, reach out to:
PS: Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.
#J-18808-Ljbffr
Health AI Conversation Evaluator in London employer: Mercor
Mercor is an exceptional employer that champions innovation and creativity in the AI sector, offering a fully remote work environment that promotes flexibility and independence. With a strong focus on employee growth, team members are encouraged to take ownership of their projects while benefiting from a supportive culture that values collaboration and continuous learning. Joining Mercor means being part of a forward-thinking company backed by industry leaders, where your contributions directly impact cutting-edge search technologies.
