Technology AI Evaluation Expert

4 weeks, 1 day ago
Contract
Mid Level
Data Science and Analytics
Weekday

Weekday

Weekday helps companies hire engineers who are vouched by other software engineers, enabling passive income for engineers. They offer services like drafting outreach messages, shortlisting candidates, and conducting reference checks. Backed by Y Combin...

Construction & Engineering
11-50
Founded 2020

Description

  • Create realistic, multi-step benchmark tasks based on professional technology workflows.
  • Develop evaluation challenges using technical specifications, architecture documents, API documentation, codebases, web research, and code execution.
  • Define clear expected outputs and objective evaluation criteria for each task.
  • Write ground-truth solutions and structured scoring rubrics.
  • Design tasks that assess reasoning, technical understanding, instruction following, and output quality.
  • Maintain high standards of technical accuracy, clarity, and reproducibility.
  • Apply domain expertise from software engineering, data science, or analytics to create authentic evaluation scenarios.
  • Collaborate with research teams to improve benchmark quality and consistency.
  • Continuously refine tasks based on project feedback and evolving evaluation requirements.

Requirements

  • Minimum 3 years of hands-on professional experience in software engineering, data science, or data analytics.
  • Strong understanding of technical documentation, software development workflows, and engineering best practices.
  • Experience working with codebases, APIs, technical specifications, or system architecture documentation.
  • Excellent analytical thinking and problem-solving skills.
  • Strong written communication with the ability to create clear technical instructions and evaluation criteria.
  • Ability to work independently while maintaining high standards of accuracy and consistency.
  • Experience designing evaluation standards, scoring rubrics, or ground-truth solutions is preferred.
  • Comfort working with technical materials such as technical specs, architecture diagrams, and workplace resources.
  • Ability to commit 15–20 hours per week.
  • Visa sponsorship is not available for this opportunity.

Benefits

  • Compensation of $60–$75 per hour.
  • Fully remote work completed on your own schedule.
  • Flexible working hours.
  • Independent contractor engagement.
  • Weekly payments processed through supported payment platforms.
  • Opportunity to work on intellectually challenging AI research projects.
  • Meaningful impact on AI benchmark quality and technical reasoning systems.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

Politics Content Reviewer (AI Evaluation)

Gramian Consultancy Group Professional Services

Gramian Consultancy is hiring a remote Political Science Domain Expert to support evaluation and improvement of large language models through domain-specific prompt creation, response review, and benchmark data development.

LLM
5 hours, 40 minutes ago

AI Trainer - Mechanical Engineers - CAD Expertise - (Remote Advisory - France)

Prolific 51-250 Professional Services

Prolific is seeking Mechanical Engineers to join its remote expert network for CAD-focused AI research projects that capture real-world design and validation workflows.

Linux Python
1 day, 4 hours ago

AI Trainer - Mechanical Engineers - CAD Expertise - (Remote Advisory - Germany)

Prolific 51-250 Professional Services

Prolific is seeking mechanical engineers to join a remote advisory talent pool supporting AI research by documenting real-world CAD design and validation workflows.

Linux Python
1 day, 4 hours ago

AI Trainer - Mechanical Engineers - CAD Expertise - (Remote Advisory - Italy)

Prolific 51-250 Professional Services

Prolific is seeking mechanical engineers to join a remote advisory talent pool supporting AI model development through CAD design challenges and validation workflows.

Linux Python
1 day, 4 hours ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers